-
Notifications
You must be signed in to change notification settings - Fork 116
feat: Add shapes.txt validator #2123
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
cswilson252
wants to merge
48
commits into
MobilityData:master
Choose a base branch
from
cswilson252:noShapesNoDrt
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 14 commits
Commits
Show all changes
48 commits
Select commit
Hold shift + click to select a range
0fcaf9e
push for diff
cswilson252 ee49572
add to own file
cswilson252 ffbcca5
add validator functionality
cswilson252 d078d3b
half baked test +some lint
cswilson252 abadac7
push up actual notice tests
cswilson252 5caa4e8
fix to asserttrue?
cswilson252 e457ddc
assertThat?
cswilson252 4ca9741
make the non-errors assertFalse
cswilson252 13e3475
assertThat
cswilson252 32a2cfd
add import
cswilson252 4d5f495
other assertThat import
cswilson252 95a83c6
get rid of assertFalse
cswilson252 337b917
isFalse
cswilson252 0e93ab1
Merge branch 'master' into noShapesNoDrt
cswilson252 6923baf
Merge branch 'master' into noShapesNoDrt
cswilson252 a1c4524
copilot feedback
cswilson252 19a3143
missing import
cswilson252 f5e6311
comparisonfailure
cswilson252 d5196e4
we have to pass another arg, luckily its nullable
cswilson252 98c14d0
remove 1f check in favour of 1
cswilson252 ce9c224
isAtLeast assertation check
cswilson252 2447088
not 1l?
cswilson252 528b409
fix semicolon
cswilson252 8329286
Merge branch 'master' into noShapesNoDrt
cswilson252 dfe1600
correct tests and validator loop logic
cswilson252 22b978b
return, don't break
cswilson252 69af53e
lint
cswilson252 04c0638
fully remove arguments
cswilson252 3763437
0 rows
cswilson252 6962ced
-1 rows
cswilson252 516dab9
break if rows are -1
cswilson252 b86fa31
try returning
cswilson252 d7de502
[Skip CI] push method signature WIP
cswilson252 e4e72a8
rely on FeedMetadata in validator method
cswilson252 7d7564d
stringify filename
cswilson252 86e77ef
pass single feedContainer
cswilson252 ccaedfd
lint
cswilson252 52b330e
add createFeedContainer()
cswilson252 049e820
lint
cswilson252 d3ba6fd
Update main/src/test/java/org/mobilitydata/gtfsvalidator/validator/Mi…
cswilson252 da570f4
forStatus
cswilson252 e1c419d
remove semicolon
cswilson252 41e6fb2
change to createShapeTable
cswilson252 390eb9e
not a forStatus?
cswilson252 de8c979
guided copilot feedContainer() logic change
cswilson252 f8aff12
lint
cswilson252 6b732c1
Merge branch 'master' into noShapesNoDrt
cswilson252 5661112
Merge branch 'master' into noShapesNoDrt
davidgamez File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
78 changes: 78 additions & 0 deletions
78
main/src/main/java/org/mobilitydata/gtfsvalidator/validator/MissingShapesFileValidator.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| package org.mobilitydata.gtfsvalidator.validator; | ||
|
|
||
| import static org.mobilitydata.gtfsvalidator.notice.SeverityLevel.WARNING; | ||
|
|
||
| import javax.inject.Inject; | ||
| import org.mobilitydata.gtfsvalidator.annotation.GtfsValidationNotice; | ||
| import org.mobilitydata.gtfsvalidator.annotation.GtfsValidationNotice.FileRefs; | ||
| import org.mobilitydata.gtfsvalidator.annotation.GtfsValidator; | ||
| import org.mobilitydata.gtfsvalidator.notice.NoticeContainer; | ||
| import org.mobilitydata.gtfsvalidator.notice.ValidationNotice; | ||
| import org.mobilitydata.gtfsvalidator.table.GtfsLocationGroupsTableContainer; | ||
| import org.mobilitydata.gtfsvalidator.table.GtfsShapeTableContainer; | ||
| import org.mobilitydata.gtfsvalidator.table.GtfsStopTime; | ||
| import org.mobilitydata.gtfsvalidator.table.GtfsStopTimeSchema; | ||
| import org.mobilitydata.gtfsvalidator.table.GtfsStopTimeTableContainer; | ||
| import org.mobilitydata.gtfsvalidator.table.GtfsTripSchema; | ||
|
|
||
| /** | ||
| * Validates that the feed has either a `shapes.txt` file, or uses zone-based DRT or fixed-stops | ||
| * DRT. | ||
| * | ||
| * <p>Generated notice: {@link MissingRecommendedFileNotice}. | ||
| */ | ||
| @GtfsValidator | ||
| public class MissingShapesFileValidator extends FileValidator { | ||
| private final GtfsShapeTableContainer shapeTable; | ||
| private final GtfsStopTimeTableContainer stopTimeTable; | ||
| private final GtfsLocationGroupsTableContainer locationGroupsTable; | ||
|
|
||
| @Inject | ||
| MissingShapesFileValidator( | ||
| GtfsShapeTableContainer shapeTable, | ||
| GtfsStopTimeTableContainer stopTimeTable, | ||
| GtfsLocationGroupsTableContainer locationGroupsTable) { | ||
| this.shapeTable = shapeTable; | ||
| this.stopTimeTable = stopTimeTable; | ||
| this.locationGroupsTable = locationGroupsTable; | ||
| } | ||
|
|
||
| @Override | ||
| public void validate(NoticeContainer noticeContainer) { | ||
| for (GtfsStopTime stopTime : stopTimeTable.getEntities()) { | ||
| String stopId = stopTime.toString(); | ||
| Boolean missingShapes = shapeTable.isMissingFile(); | ||
| Boolean hasLocationId = stopTimeTable.hasColumn("location_id"); | ||
| Boolean hasLocationGroupId = stopTimeTable.hasColumn("location_group_id"); | ||
| Boolean hasLocationGroupsRecord = locationGroupsTable.isParsedSuccessfully(); | ||
|
cswilson252 marked this conversation as resolved.
Outdated
|
||
| // Do we not have a shapes.txt file and not have a location_id (required for Zone-Based DRT)? | ||
| if (missingShapes && !hasLocationId) { | ||
| // Do we not have a record in location_groups.txt and not have a trip in stop_times.txt that | ||
| // references location_group_id (required for Fixed-Stop DRT)? | ||
| if (!hasLocationGroupsRecord && !hasLocationGroupId) { | ||
|
cswilson252 marked this conversation as resolved.
Outdated
|
||
| noticeContainer.addValidationNotice( | ||
| new MissingRecommendedFileNotice(stopTime.csvRowNumber(), stopId)); | ||
| } | ||
|
cswilson252 marked this conversation as resolved.
Outdated
cswilson252 marked this conversation as resolved.
Outdated
|
||
| } | ||
| } | ||
| } | ||
|
|
||
| /** | ||
| * A feed must have a `shapes.txt` file, and/or use zone-based or fixed-stops DRT to be usable. | ||
| */ | ||
| @GtfsValidationNotice( | ||
| severity = WARNING, | ||
| files = @FileRefs({GtfsStopTimeSchema.class, GtfsTripSchema.class})) | ||
| static class MissingRecommendedFileNotice extends ValidationNotice { | ||
| /** The row number of the faulty record. */ | ||
|
cswilson252 marked this conversation as resolved.
Outdated
|
||
| private final int csvRowNumber; | ||
|
|
||
| /** The faulty record's id. */ | ||
| private final String tripId; | ||
|
|
||
| MissingRecommendedFileNotice(int csvRowNumber, String tripId) { | ||
| this.csvRowNumber = csvRowNumber; | ||
| this.tripId = tripId; | ||
| } | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
112 changes: 112 additions & 0 deletions
112
...rc/test/java/org/mobilitydata/gtfsvalidator/validator/MissingShapesFileValidatorTest.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,112 @@ | ||
| package org.mobilitydata.gtfsvalidator.validator; | ||
|
|
||
| import static com.google.common.truth.Truth.assertThat; | ||
|
|
||
| import java.util.ArrayList; | ||
| import java.util.List; | ||
| import org.junit.Test; | ||
| import org.mobilitydata.gtfsvalidator.notice.NoticeContainer; | ||
| import org.mobilitydata.gtfsvalidator.notice.ValidationNotice; | ||
| import org.mobilitydata.gtfsvalidator.table.GtfsLocationGroups; | ||
| import org.mobilitydata.gtfsvalidator.table.GtfsLocationGroupsTableContainer; | ||
| import org.mobilitydata.gtfsvalidator.table.GtfsShape; | ||
| import org.mobilitydata.gtfsvalidator.table.GtfsShapeTableContainer; | ||
| import org.mobilitydata.gtfsvalidator.table.GtfsStopTime; | ||
| import org.mobilitydata.gtfsvalidator.table.GtfsStopTimeTableContainer; | ||
|
|
||
| public class MissingShapesFileValidatorTest { | ||
|
|
||
| private static List<GtfsShape> createShapeTable(int rows) { | ||
| ArrayList<GtfsShape> shapes = new ArrayList<>(); | ||
| for (int i = 0; i < rows; i++) { | ||
| shapes.add(new GtfsShape.Builder().setCsvRowNumber(i + 1).setShapeId("s" + i).build()); | ||
| } | ||
| return shapes; | ||
|
cswilson252 marked this conversation as resolved.
|
||
| } | ||
|
|
||
| private static List<GtfsStopTime> createStopTimesTable( | ||
| int rows, String locationGroupId, String locationId) { | ||
| ArrayList<GtfsStopTime> stopTimes = new ArrayList<>(); | ||
| for (int i = 0; i < rows; i++) { | ||
| stopTimes.add( | ||
| new GtfsStopTime.Builder() | ||
| .setCsvRowNumber(i + 1) | ||
| .setLocationGroupId(locationGroupId) | ||
| .setLocationId(locationId) | ||
| .build()); | ||
| } | ||
|
cswilson252 marked this conversation as resolved.
Outdated
|
||
| return stopTimes; | ||
| } | ||
|
|
||
| private static List<GtfsLocationGroups> createLocationGroupsTable( | ||
| int rows, String groupId, String groupName) { | ||
| ArrayList<GtfsLocationGroups> locationGroups = new ArrayList<>(); | ||
| for (int i = 0; i < rows; i++) { | ||
| locationGroups.add( | ||
| new GtfsLocationGroups.Builder() | ||
| .setCsvRowNumber(i + 1) | ||
| .setLocationGroupId(groupId) | ||
| .setLocationGroupName(groupName) | ||
| .build()); | ||
| } | ||
| return locationGroups; | ||
| } | ||
|
|
||
| @Test | ||
| public void testShapesFileAndFixedDrtPresent() { | ||
| List<ValidationNotice> notices = | ||
| generateNotices( | ||
| createShapeTable(1), | ||
| createStopTimesTable(1, "a", null), | ||
| createLocationGroupsTable(1, "b", "testgroup")); | ||
| boolean found = | ||
| notices.stream() | ||
| .anyMatch( | ||
| notice -> | ||
| notice instanceof MissingShapesFileValidator.MissingRecommendedFileNotice); | ||
| assertThat(found).isFalse(); | ||
| } | ||
|
|
||
| @Test | ||
| public void testShapesFileAndZoneBasedDrtPresent() { | ||
| List<ValidationNotice> notices = | ||
| generateNotices( | ||
| createShapeTable(1), | ||
| createStopTimesTable(1, null, "c"), | ||
| createLocationGroupsTable(1, "d", "t3stgroup")); | ||
| boolean found = | ||
| notices.stream() | ||
| .anyMatch( | ||
| notice -> | ||
| notice instanceof MissingShapesFileValidator.MissingRecommendedFileNotice); | ||
| assertThat(found).isFalse(); | ||
| } | ||
|
|
||
| @Test | ||
| public void testNoShapesFileAndNoDrtPresent() { | ||
| List<ValidationNotice> notices = | ||
|
cswilson252 marked this conversation as resolved.
|
||
| generateNotices( | ||
| createShapeTable(0), | ||
| createStopTimesTable(1, null, null), | ||
| createLocationGroupsTable(0, null, null)); | ||
|
cswilson252 marked this conversation as resolved.
Outdated
|
||
| boolean found = | ||
| notices.stream() | ||
| .anyMatch( | ||
| notice -> | ||
| notice instanceof MissingShapesFileValidator.MissingRecommendedFileNotice); | ||
| assertThat(found).isFalse(); | ||
|
cswilson252 marked this conversation as resolved.
Outdated
|
||
| } | ||
|
|
||
| private static List<ValidationNotice> generateNotices( | ||
|
cswilson252 marked this conversation as resolved.
|
||
| List<GtfsShape> shapes, | ||
| List<GtfsStopTime> stopTimes, | ||
| List<GtfsLocationGroups> locationGroups) { | ||
| NoticeContainer noticeContainer = new NoticeContainer(); | ||
| new MissingShapesFileValidator( | ||
| GtfsShapeTableContainer.forEntities(shapes, noticeContainer), | ||
| GtfsStopTimeTableContainer.forEntities(stopTimes, noticeContainer), | ||
| GtfsLocationGroupsTableContainer.forEntities(locationGroups, noticeContainer)) | ||
| .validate(noticeContainer); | ||
| return noticeContainer.getValidationNotices(); | ||
| } | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.