feat: Add shapes.txt validator#2123
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new validator intended to warn when shapes.txt is missing unless the feed appears to use zone-based or fixed-stop DRT (per #1792).
Changes:
- Introduces
MissingShapesFileValidatorand a corresponding test class. - Adds unit tests covering shapes present + DRT present scenarios (and a no-shapes/no-DRT scenario).
- Adds an extra import in
ShapeUsageValidator.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 9 comments.
| File | Description |
|---|---|
| main/src/main/java/org/mobilitydata/gtfsvalidator/validator/MissingShapesFileValidator.java | New multi-file validator for warning on missing shapes.txt unless DRT indicators are present. |
| main/src/test/java/org/mobilitydata/gtfsvalidator/validator/MissingShapesFileValidatorTest.java | New unit tests for the validator behavior. |
| main/src/main/java/org/mobilitydata/gtfsvalidator/validator/ShapeUsageValidator.java | Adds an import for a nested notice type. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
i'm a bit stuck as to why the test failure is happening |
…ssingShapesFileValidatorTest.java Co-authored-by: David Gamez <1192523+davidgamez@users.noreply.github.com>
|
yippee! ready for review, thanks again! :) |
📝 Acceptance Test Report📋 Summary❌ The rule acceptance test has failed for commit ca15e80 📊 Notices ComparisonNew Errors (0 out of 1001 datasets, ~0%) ✅No changes were detected due to the code change. Dropped Errors (0 out of 1001 datasets, ~0%) ✅No changes were detected due to the code change. New Warnings (48 out of 1001 datasets, ~5%) ❌Details of new errors due to code change, which is above the provided threshold of 1%.
Dropped Warnings (0 out of 1001 datasets, ~0%) ✅No changes were detected due to the code change. New Info Notices (0 out of 1001 datasets, ~0%) ✅No changes were detected due to the code change. Dropped Info Notices (0 out of 1001 datasets, ~0%) ✅No changes were detected due to the code change. 🛡️ Corruption Check2 out of 1003 sources (~0 %) are corrupted.
🔍 System errors for mdb-1114 (reference)[
{
"code": "i_o_error",
"severity": "ERROR",
"totalNotices": 1,
"sampleNotices": [
{
"exception": "java.util.zip.ZipException",
"message": "Archive is not a ZIP archive"
}
]
}
]🔍 System errors for mdb-1114 (latest)[
{
"code": "i_o_error",
"severity": "ERROR",
"totalNotices": 1,
"sampleNotices": [
{
"exception": "java.util.zip.ZipException",
"message": "Archive is not a ZIP archive"
}
]
}
]🔍 System errors for mdb-1123 (reference)[
{
"code": "i_o_error",
"severity": "ERROR",
"totalNotices": 1,
"sampleNotices": [
{
"exception": "java.util.zip.ZipException",
"message": "Archive is not a ZIP archive"
}
]
}
]🔍 System errors for mdb-1123 (latest)[
{
"code": "i_o_error",
"severity": "ERROR",
"totalNotices": 1,
"sampleNotices": [
{
"exception": "java.util.zip.ZipException",
"message": "Archive is not a ZIP archive"
}
]
}
]💾 Out of Memory CheckNo datasets experienced an OutOfMemoryError. ⏱️ Performance Assessment📈 Validation TimeAssess the performance in terms of seconds taken for the validation process.
📜 Memory Consumption
|
|
@skalexch Tagging you here to review the acceptance tests |
skalexch
left a comment
skalexch
left a comment
There was a problem hiding this comment.
@emmambd @davidgamez I did a query to check for the flagged feeds. Only one "seemed" to have Fixed Stops DRT. I double checked it and it does not have trips whatsoever so it's a false flag.
A second query is regarding which feeds from the subset should fail the tests but did not. So for some reason, there are more than 120 missing warnings.
|
@skalexch mdb-2237 doesn't have fixed stops demand responsive transit on the Mobility Database, so I'm wondering if that's an acceptance test issue rather than an issue with the PR logic? As for false negatives, after a few spot checks I suspect it's due to the shapes.txt file check. I see several cases where shapes.txt exists but only the column headers are populated and nothing else, as well as a few cases where the file does not exist at all. cc @davidgamez @cswilson252 |
|
@emmambd for mdb-2237 the acceptance tests and the MobilityDatabase are both accurate in saying that it does not have any of the 3 features. My query shows it associated with Fixed-Stops DRT so that may be a previous validation before we refined the logic of feature detection, so that's not an issue. But overall, the logic of the validation needs to be refined to make sure all feeds are detected. @cswilson252 I don't think there is a missing condition in the code, my assumption is that maybe one of the boolean variables is returning a false positive. In the validator code, I have not seen an expression like |
I would keep |
|
hi all, I will add |
Summary:
Add validator that checks if either a shapes.txt and/or a fixed or zone-based DRT service is present (as per #1792)
DRT functionality added as per this reference
Fixes #1792
Expected behavior:
Validator will print WARNING if shapes.txt and signs of a DRT feature are not found.
Please make sure these boxes are checked before submitting your pull request - thanks!
gradle testto make sure you didn't break anything