Proposal: Add MODIFIED and pickup and drop-off types to GTFS-RT; experimentally deprecate SKIPPED

[ericouyang]

Background

To remedy operational challenges, transit providers dynamically change the level of service provided by a vehicle for a given trip. For example, for a vehicle that is running late, a dispatcher may instruct a vehicle to go to "Drop Off Only" to regain time when running late or address a gap in service.

Today, in GTFS-RT, there is only the ability to indicate a stop as entirely SKIPPED. However, there is not a way to indicate that a stop is going to be partially served, such as stopping there upon request to let a passenger disembark and not picking up additional passengers in the scenario above.

Proposal

This proposal adds support for a more detailed relationship of a vehicle to a scheduled stop beyond skipping it completely and brings GTFS-realtime into consistency with GTFS-static's stop_times.txt. It ​builds on a recent addition (#219) to support real-time changes using StopTimeProperties and adds pickup_type and drop_off_type as experimental fields. GTFS static already supports specifying pickup_type and drop_off_type as a part of stop_times.txt. Those existing definitions are leveraged and referenced as a part of this change.

To avoid an ambiguous state where the currently supported SKIPPED scenario could also be alternatively represented as drop_off_type=NO_DROP_OFF and pickup_type=NO_PICKUP using StopTimeProperties, this proposal experimentally deprecates SKIPPED in favor of that more explicit representation. When StopTimeProperties are modified in a way that impacts the rider experience, now a new enumeration, MODIFIED, will be used instead.

Both the gtfs-realtime.proto file and documentation have been updated accordingly. Additionally, like for DUPLICATED, a migration guide has been provided to detail a plan for a smooth transition for existing producers and consumers.

This pull request is a subset of and based on the GTFS-ServiceChanges v3.1 spec:
https://bit.ly/gtfs-service-changes-v3_1

This proposal builds on prior art from @lionel-nj and @barbeau (see MobilityData#47)

[skinkie]

This is a disaster waiting to happen. Either we allow for the exchange of fully replaceable routes including the network, or we disallow network changes, but this balance on what sort of could work for some "big players" does not make any sense.

[barbeau]

Also note that the proposed stop_time_properties.pickup_type = NO_PICKUP is effectively the same as the existing vehicle.occupancy_status = NOT_ACCEPTING_PASSENGERS. Should we deprecate OccupancyStatus.NOT_ACCEPTING_PASSENGERS as part of this proposal as well? Or do people view them as valuable individually (in a VP vs. TU feed)?

[ericouyang]

@skinkie - Could you elaborate a bit more in terms of the risks you're seeing with a piecemeal approach?

While I definitely agree that it would be great to implement more of the proposal in GTFS-ServiceChanges V3.1 to support complex route changes, I believe that there's still value in tackling it piece-by-piece (like in the scenario described above). This PR follows the precedent set by #219 and #221.

[ericouyang]

Should we deprecate OccupancyStatus.NOT_ACCEPTING_PASSENGERS as part of this proposal as well? Or do people view them as valuable individually (in a VP vs. TU feed)?

@barbeau - Good question! Since one is describing the vehicle in real-time vs. the other describing a trips' relationship with upcoming stops, I view them as distinct but related. I could see value in terms of adding an inline clarification about how they can be used together, such as:

If NO_PICKUP is set because the vehicle is at capacity and no longer accepting additional boardings, vehicle.occupancy_status = NOT_ACCEPTING_PASSENGERS should also be set.

There's also the issue of determining how many stops to set NO_PICKUP for while a vehicle is NOT_ACCEPTING_PASSENGERS. Would it make sense to set it for all upcoming stops until the vehicle has an occupancy status that supports boarding again?

On a practical level, I also wonder how often NOT_ACCEPTING_PASSENGERS is used by feed creators, how it's displayed in passenger apps, and if it has any ramifications on the display of predictions, like the way that Alert.effect = NO_SERVICE will cause predictions to be suppressed. If it's not actively used, no one has plans to implement it and/or there's no special treatment that distinguishes it from FULL, that could be a reason to remove it in favor of an alternative approach.

[skinkie]

@skinkie - Could you elaborate a bit more in terms of the risks you're seeing with a piecemeal approach?

Because this is a moving target from a software architecture perspective. If the target is replacing full trips: standardize that.

It is impossible to keep up to date with partial implementations, unless you you already support all that, and are waiting for a full implementation that is being stopped by incremental implementations.

While I definitely agree that it would be great to implement more of the proposal in GTFS-ServiceChanges V3.1 to support complex route changes, I believe that there's still value in tackling it piece-by-piece (like in the scenario described above). This PR follows the precedent set by #219 and #221.

Does not mean that the precedent has resulted in global support for it.

[ericouyang]

If the target is replacing full trips: standardize that.

@skinkie Thanks for your thoughtful response. At this time, our target is not replacing full trips or being able to create arbitrary ad-hoc trips on the fly (DUPLICATED, as previously adopted, is in scope). As noted in the original description, our intent at this time is to support on-the-fly changes to partially cancel service, such as to have a vehicle go drop-off only. The spec change process explicitly discourages "speculative" features and this particular change, as noted, is based on real-world customer needs.

Because this is a moving target from a software architecture perspective.

It may be worth keeping in mind the full scope of GTFS-ServiceChanges V3.1 as support is developed for parts of it. I understand the frustration of a moving target and I encourage you (and others) to propose more holistic changes to the spec, if those are in-fact being implemented.

[skinkie]

-1 (OpenGeo)
Pro implementing this as complete solution, against doing this as experimental stuff just causing fragmentation.

[ericouyang]

I've called for a vote for adoption of this experimental future on the Google Group (https://groups.google.com/g/gtfs-realtime/c/YWY9IoMQF7g).

Please vote with a +1 (in favor) or -1 (against) before Thursday, April 8th at 23:59:59 UTC. Thanks!

[ericouyang]

Since we didn't get much feedback last time, I've called for a revote for adoption of this experimental future on the Google Group (https://groups.google.com/g/gtfs-realtime/c/YWY9IoMQF7g?pli=1).

Please vote with a +1 (in favor) or -1 (against) before Wednesday, Aug 25th at 23:59:59 UTC. Thanks!

[gcamp]

I think the version problem (can you really have experimental versions?) at the very least needs to be addressed before an other vote.

I would personally not increase the version considering it has a bigger meaning and the proposal is only experimental.

You can still maintain two versions side by side with other means like different URLs.

[barbeau]

I agree with @gcamp. Experimental versions isn't something in the current CHANGES doc and specifying how that works is probably it's own proposal.

The main thing is that feels a bit breaking is that in an existing feed without these fields, SKIPPED could be appropriate for the drop-off only scenario as the behavior for a rider-facing application we might expect is to not show predictions for NO_PICKUP. With these feeds available, we're imagining that we would keep the relationship SCHEDULED since the vehicle may stop, but only upon rider request.

I think this could be handled via a migration guide like we did for DUPLICATED?

These conditions should be clarified somewhere so it's clear to producers and consumers how the data will be interpreted.

[ericouyang]

Thanks, @gcamp and @barbeau! I've gone ahead and reverted the version bump change and will take a first crack at documenting a migration guide for this experimental addition.

[google-cla]

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

[paulswartz]

+1 @mbta

[paulswartz]

@googlebot I fixed it.

[skinkie]

I would still prefer to keep SKIPPED (implying both drop-off is false) for the situation that the stop is not passed through.

[barbeau]

I would still prefer to keep SKIPPED (implying both drop-off is false) for the situation that the stop is not passed through.

@skinkie Doesn't this approach have backwards compatibility issues though? If producers are currently using SKIPPED to describe the situation where a vehicle passes though a stop (but does not stop), which is allowed by the current spec, going forward how do we know which producers intend the new definition (stop is not passed through) vs. the original definition (stop is passed through but vehicle doesn't stop)?

[barbeau]

Should this proposal itself be deprecated or later removed, I think the cleanest approach then would be to un-deprecate SKIPPED to signal that as the recommended approach again for this scenario.

I agree with this - we should probably add an explicit note on that in the .proto comments on SKIPPED. There shouldn't be an issue with this in terms of protobuf compatibility of existing producers/consumers.

If the experimental feature becomes officially adopted, I think that point would be an appropriate time to change SKIPPED to reserved to prevent further usage.

So I know removing an enum value from the .proto will break new consumers of existing feeds that are producing the value (this happened with REPLACEMENT in the past).

However, I believe using reserved on enum values does not have this issue as the new consumer bindings still recognize the previous existence of the enum value and therefore don't throw a (not a valid enum value) error. My understanding is that reserved just prevents you from using the enum in your code the next time you regenerate your bindings from the .proto (i.e., your code won't compile if you try to use the enum value). Conversely, for enum values marked [deprecated=true] you can still use the enum value in your code and it will compile, but it will give you warnings that it's deprecated.

Unfortunately the exact mechanics and backwards compatibility issues with reserved aren't well-documented in the Protocol Buffer documentation, so it would be good to get feedback from someone with more experience managing backwards compatibility issues in protobufs or test ourselves to confirm before using reserved.

[skinkie]

I would still prefer to keep SKIPPED (implying both drop-off is false) for the situation that the stop is not passed through.

@skinkie Doesn't this approach have backwards compatibility issues though? If producers are currently using SKIPPED to describe the situation where a vehicle passes though a stop (but does not stop), which is allowed by the current spec, going forward how do we know which producers intend the new definition (stop is not passed through) vs. the original definition (stop is passed through but vehicle doesn't stop)?

Functionally the effect for the traveler is the same. The only difference may be that the vehicle location may follow the expected path, while the Skipped-message suggest it may follow unexpected path.

[SteveGladstone]

Definitely have a concern as a producer (via Trapeze/Swiftly/others) over SKIPPED being deprecated. Uncertainty over how the reserved keyword in the proto behaves makes me cautious. The last thing we as an agency (Maryland Transit) want is to be required to go through a costly upgrade process. And as a consumer of feeds for use in other applications, SKIPPED plus NOT_ACCEPTING_PASSENGERS combined already covers things well enough by letting riders waiting know their stop won't be served because the bus is not accepting passengers. Even if it's not explicitly what's intended, it fill the gap conceptually.

That being said, the granularity of explaining what's happening with service in travel can be powerful and important. But it's a bit unclear how this would get reflected for riders IMO.

One example - train service that will skip stops to make up time due to would go from SKIPPED to pickup_type=NO_PICKUP and dropoff_type=NO_DROP_OFF. Without a corresponding Alert with effect=NO_SERVICE, what should this look like on a website, mobile app, or other digital medium? And given the vehicle.occupancy_status=NOT_ACCEPTING_PASSENGERS enum, should that be triggered as well? I agree with Eric that Vehicle and Trip data is unique, though there is overlap and consistency across all feeds would be preferred. As far as I'm aware (?), there's still a discussion around occupancy_status, occupancy_percentage, and crowding_levels to reflect passenger loads and associated effects. If a vehicle has passengers but is not accepting passengers, what should this look like in such a way as to not cause confusion amongst riders?

Another example that could make this a little more difficult. Consider bus service with stops potentially every 1-2 minutes (inefficient, but humor me). How should this change be reflected by producers when vehicles go in and out of a NO_PICKUP state? A bus may get too crowded, go into NO_PICKUP for 3 stops to a transfer hub, resume pickup for 3 stops, then have to go NO_PICKUP after stop 4 because it's another transfer hub. The oscillation between states seems like it would be problematic- though this is as much an agency policy issue as it would be a feed consumer issue. If the state is based on VLU logic tied to passenger load and able to be manually triggered, you could give riders a headache over whether their bus will or will not let them board.

Thus a question to the group - how should this info by communicated to riders and how should feed consumers utilize it?

[ericouyang]

Functionally the effect for the traveler is the same.

@skinkie - I respectfully disagree about this and share @barbeau's concern around backwards compatibility. Upon a re-read of the documentation for SKIPPED, making a semantic change does impact how the values are interpreted as "Delay from a previous stop in the trip does propagate over the SKIPPED stop." I interpret that as meaning, by default, that a vehicle isn't performing, say a detour that would further the delay more than previously indicated.

If you think it's a valuable distinction between vehicles that pass through a stop vs. ones that do not, I would suggest that the best way to do that is introducing another enum (such as DETOURED) with explicit semantics, perhaps with a requirement of specifying a value for delay or otherwise requiring a StopTimeEvent at the next possible StopTimeUpdate. However, I don't think this particularly useful of a concept in TripUpdate. Maybe it could be an extension to VehicleStopStatus instead?

[ericouyang]

@barbeau, @SteveGladstone - Regarding reserved, upon further reflection, I think how we choose to remove SKIPPED, if at all, can be a decision for the future. I've added new language into the spec to reflect interim guidance. The decision between keeping it as deprecated (and then removing the additional experimental language) or reserved to me primarily rests on how strict we want to be in terms of fully removing SKIPPED from the ecosystem over time vs. more of a softer encouragement.

[ericouyang]

The last thing we as an agency (Maryland Transit) want is to be required to go through a costly upgrade process.

@SteveGladstone - We totally share the same sentiment and, as a feed producer, intend to allow for agencies to decide if and when they'd like to exercise the migration notification process documented in migration-modified.md. The choice to use this and other experimental features defined by GTFS-ServiceChanges v3 is fully opt-in and based on the needs of agencies.

Regarding NOT_ACCEPTING_PASSENGERS, while there's certainly overlaps in scenarios, I see that and the other discussion around representing occupancy as related, but separate concept. A vehicle may be pickup_type=NO_PICKUP and drop_off_type=NO_DROP_OFF purely due to running late, but have occupancy_status=EMPTY. Personally, I'm not sure how often NOT_ACCEPTING_PASSENGERS is used today and how helpful it is as a distinct concept from FULL, especially, as you noted, there can be frequent changes in status as a vehicle continue to load/unload passengers (particularly if there's driver discretion), so by the time a vehicle actually arrives at a stop, the status at a prior point in time may not be valid and cause rider confusion.

With respect to how this appears to the passenger, I'd love to get any insights from @gcamp with how pickup_type and drop_off_type from static GTFS are handled today within Transit. I would presume that the behavior for these setting these in real-time to be the same as what already would happen today. The key intent behind the changes proposed here is not erroneously misleading passengers that they could board an incoming vehicle, but while in a routing/navigation mode, the estimated arrival time could still leverage the predicted timing.

Consider bus service with stops potentially every 1-2 minutes (inefficient, but humor me). How should this change be reflected by producers when vehicles go in and out of a NO_PICKUP state? A bus may get too crowded, go into NO_PICKUP for 3 stops to a transfer hub, resume pickup for 3 stops, then have to go NO_PICKUP after stop 4 because it's another transfer hub. The oscillation between states seems like it would be problematic- though this is as much an agency policy issue as it would be a feed consumer issue. If the state is based on VLU logic tied to passenger load and able to be manually triggered, you could give riders a headache over whether their bus will or will not let them board.

Our intended use for these new properties is for an agency-triggered intervention to help restore service, such as when there is bunching, rather than one that's automated based on any signals from a bus (so very different than setting occupancy_status based on APCs). The status communicated to consumers of the feed would therefore respect decisions made by a dispatcher in operations. From our research, it appears that usually there is a point along the route (often until the end of the current trip) where a drop-off only would apply, so it wouldn't oscillate and would likely be overly taxing of a process for a dispatcher to change the status frequently. That said, one could also imagine a driver-initiated flow through some onboard interface to self-report statuses and a policy could be derived to, say, limit that propagation to X stops into the future. As a rider, I'd say that I would probably prefer the pleasant surprise of a vehicle arriving earlier than expected while waiting at a stop for a further out vehicle rather than a vehicle being predicted to arrive but not being able to board. Hence, I think real-time passenger information software may want to adopt a more conservative stance to tend to mark more stops as NO_PICKUP should an explicit end of a drop-off only not be specified.

[sam-hickey-arcadis]

@ericouyang Thanks for addressing our comments! The “experimentally deprecated” language added in a few places looks nice.

We still have a concern about marking SKIPPED as [deprecated=true] while the new pickup_type and drop_off_type are experimental. The protocol buffers documentation says the following about deprecated:

deprecated (field option): If set to true, indicates that the field is deprecated and should not be used by new code.

This is in conflict with the "experimentally deprecated" language that says, for example, "In this interim period, users may continue to use this value for compatability." Our preference is to not mark the SKIPPED field as [deprecated=true] at this point and instead keep the "experimentally deprecated" language for now. We're happy to see this all adopted experimentally, but we still have concerns about the migration steps required from a practical perspective that will need to be addressed before moving out of the experimental phase such that the change doesn't leave behind existing producers or consumers of SKIPPED.

Also, we share the concern noted by others of changing SKIPPED to mean a stop is not passed through.

[colemccarren]

Also, we share the concern noted by others of changing SKIPPED to mean a stop is not passed through.

In my mind, since the spec defines SKIPPED without deliberate mention of route shape adherence, it wouldn't be unreasonable to assume different consumers and producers read the intended use in different ways. It'd be a shame if that ambiguity halts progress on a MODIFIED ScheduleRelationship value, which gives agencies dramatically more flexibility in communicating to riders.

Regarding NOT_ACCEPTING_PASSENGERS, while there's certainly overlaps in scenarios, I see that and the other discussion around representing occupancy as related, but separate concept. A vehicle may be pickup_type=NO_PICKUP and drop_off_type=NO_DROP_OFF purely due to running late, but have occupancy_status=EMPTY.

I absolutely agree - here's another scenario. Having driven in a college town, there were many nights where something... stinky... happened on a bus, and agency policy stated that vehicles must switch their destination sign to "Drop Off Only" until empty, when they could then return to the bus yard and be cleaned by someone with the appropriate equipment. This is yet another distinct scenario, completely independent of a vehicle's occupancy status, where an agency might want to define (individual and downstream) stop pickup and drop-off behavior, without necessarily purging all passenger-facing real time data for a vehicle (for example, in the above scenario, persons with disabilities using applications that utilize agency real-time feeds that are still sitting on the bus - the data can still assist them in when to request their drop-off).

The ability to do this, and do it in a way that mirrors the (already understood) static pickup and drop-off types is really exciting. If experimental deprecation is a valid way to achieve this for the spec, without forcing (or even encouraging) agencies to stop using SKIPPED, awesome. If not, would it make sense for the definition of an experimental MODIFIED value to include instructions to specifically not use SKIPPED elsewhere in the same .proto? For the agencies that are interested in using MODIFIED to describe what some agencies are just using SKIPPED for, but with the additional capabilities?

[SteveGladstone]

The ability to do this, and do it in a way that mirrors the (already understood) static pickup and drop-off types is really exciting.

I concur. Having the flexibility to handle these scenarios would be awesome. I think the sticky point right now is SKIPPED with deprecation and backwards compatibility given some of us with older CAD/AVL systems might need 1-3 years for "migration" (ug).

Also curious if @gcamp can provide any insight on how Transit might reflect this.

[ericouyang]

The voting period for this proposal has ended, with one +1 and one -1 in this most recent iteration. So far, a total of 3 members have voted a +1 for some version of this proposal and 1 member has voted a -1.

As a next step, I'll continue to make further refinements to this proposal and, once it appears that there's some consensus, call for another vote.

@sam-hickey-ibigroup - That perspective totally makes sense! I've gone ahead and removed the formal deprecation tag and left the "experimentally deprecated" language for a future vote.

Also, we share the concern noted by others of changing SKIPPED to mean a stop is not passed through.

To clarify, while there was prior discussion around changing the definition, this proposal does not propose any changes to the definitions of any existing enums. As previously shared, if such a notion were helpful (I do not personally think so), I think that should be a completely separate proposal to, say, add a new enum with a new meaning. From the discussion so far, it is only @skinkie who is seeking such a change.

would it make sense for the definition of an experimental MODIFIED value to include instructions to specifically not use SKIPPED elsewhere in the same .proto?

@colemccarren - I like this idea a lot and will add some language reflecting this guidance!

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[skinkie]

Keep open.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This pull request has been closed due to inactivity. Pull requests can always be reopened after they have been closed. See the Specification Amendment Process.