Deprecate TripUpdate.schedule_relationship = ADDED, add TripUpdate.schedule_relationship = NEW / REPLACEMENT to specify new / replaced trips which do not run on a schedule from the GTFS static.

[miklcct]

The use of TripUpdate.schedule_relationship = ADDED was unspecified and different producers / consumers used it in different ways. For example, it is sometimes used to specify additional departures on an existing route, but it is also used to specify departures which can't be matched to any existing trips.

This PR attempts to fully deprecate ADDED, and introduce NEW and REPLACEMENT, based on the implementation of OpenTripPlanner which specifies the whole journey to be added or replaced. Additional fields, such as headsigns, and pickup / drop-off types, are introduced as required to support the full specification of completely new trips.

NEW

In this proposal, TripUpdate.schedule_relationship = NEW should be used to add trips which do not duplicate an existing trip. Such trips are considered to be unrelated to any existing trips in the GTFS Static and can serve an arbitrary pattern, including completely new patterns not found in the GTFS Static.

A typical use case is for relief trips for extra demand, typically after big events.

NEW trips are intended as a migration target for existing feeds which use ADDED to specify new trips unrelated to the static GTFS, including OpenTripPlanner. trip_id in the TripDescriptor for new trips must be completely new (not found in GTFS static) and unique, and a start_date should also be specified as well (I am not using the word "must" here because it is permitted not to specify start_date to match scheduled trips, in this case the trip is assumed to run today).

The whole journey of the added trip must be specified, in stop order, as StopTimeUpdates inside the TripUpdate without any omission. Fields are added to TripProperties and StopTimeProperties for esssential information such as names, headsigns, pickup / drop off types.

REPLACEMENT

I propose to un-deprecate TripUpdate.schedule_relationship = REPLACEMENT as well. It works in the same way as NEW, apart from that the TripDescriptor must match one instance of a scheduled trip (like other values of TripUpdate.schedule_relationship), and that instance is replaced with the complete replacement trip specified in form of StopTimeUpdates like an added trip. The original stop times in the GTFS static are not considered by the replacement trip in any form to avoid confusion. The replacement trip can serve an arbitrary pattern with an arbitrary schedule, the only expectation is that the passenger should associate the replacement trip to actually be a replacement of the original trip.

A typical use case is for short-term timetable change, or short-term (near real-time) diversion, where the fact that the trip_id remains the same can be used by journey planners to notify the user that the booked service has been changed. (In particular, I have successfully used this feature to handle real-time train diversions in GB in OpenTripPlanner and route users to alight at diverted stops, which is something neither Google Maps nor Citymapper can do now)

This is the behaviour implemented in OpenTripPlanner, which is equivalent to deleting the matched trip, and processing the replacement TripUpdate as an ADDED trip mentioned above.

Relationship to TripModification

TripModification provides a way to modify trips en-masse by specifying a list of trip IDs where the same detour can be applied. However, it is not suited to change the schedule on a per-trip basis, replacing the trip with a completely different schedule after any diversions with different running times (common due to pathing constraints on railways).

It should be forbidden to modify the same trip via a REPLACEMENT trip update and also via a TripModification.

[gcamp]

Nice to see some movement in that direction!

I think you used a markdown editor that changed formatting on a lot of tables which makes it hard to see the actual diff from your proposal. Would it be possible to fix that?

You put a lot in the PR description that's not actually in the proposed changes. Is that just to start the discussion? Some of it is quite consequential, like the whole journey of the added trip must be specified.

I'm a bit puzzled on how a consumer is supposed to ingest ADDED changes like this with arbitrary trips with no more information than an headsign. Which route is that on? Is those added trips supported only on existing routes in the GTFS? If the answer is no, we're getting quite close to the service change proposal : https://bit.ly/gtfs-service-changes-v3_1

[leonardehrenfried]

Thanks for opening this PR!

OTP has had an implementation of ADDED for a long time but its behaviour is severely underspecified. I'd love to formalise it.

Yes, OTP allows you to create completely new free form trips that have no relation to an existing pattern or trip. It tries match the given route id to an existing one but if none is in the message a dummy one is created. For once, OTP is really following the "just give us what you have, and we will try to work it out" strategy.

The only requirement we have is that the stop ids must match the static GTFS. The question is what should happen when they don't. Should the entire update be dropped or individual stops? Does that even need to be specified?

I agree with what @gcamp said about the markdown tables and the issue description.

Lastly, you might find it easier to get this through review if you split it into two PRs: one for ADDED and one for REPLACEMENT. That's just a guess though.

[miklcct]

I think that the requirement for the whole trip to be specified is written in the code. Let me know if it is not clear enough.

I'll fix the formatting later today.

[miklcct]

Nice to see some movement in that direction!

I think you used a markdown editor that changed formatting on a lot of tables which makes it hard to see the actual diff from your proposal. Would it be possible to fix that?

You put a lot in the PR description that's not actually in the proposed changes. Is that just to start the discussion? Some of it is quite consequential, like the whole journey of the added trip must be specified.

I'm a bit puzzled on how a consumer is supposed to ingest ADDED changes like this with arbitrary trips with no more information than an headsign. Which route is that on? Is those added trips supported only on existing routes in the GTFS? If the answer is no, we're getting quite close to the service change proposal : https://bit.ly/gtfs-service-changes-v3_1

"The whole journey of the added trip must be specified" is a fact in the core of this PR, noted in the updated definition of StopTimeUpdate:

Updates to StopTimes for the trip (both future, i.e., predictions, and in some cases, past ones, i.e., those that already happened). The updates must be sorted by stop_sequence, and apply for all the following stops of the trip up to the next specified stop_time_update.
If trip.schedule_relationship is SCHEDULED, at least one stop_time_update must be provided for the trip.
If trip.schedule_relationship is ADDED or REPLACEMENT, stop_time_updates must be provided for all stops in the added or replacement trip, and the stop times in the static GTFS are not used.
If the trip is canceled or deleted, no stop_time_updates need to be provided. If stop_time_updates are provided for a canceled or deleted trip then the trip.schedule_relationship takes precedence over any stop_time_updates and their associated schedule_relationship. If the trip is duplicated, stop_time_updates may be provided to indicate real-time information for the new trip.

The route and direction of the ADDED trip is specified in TripDescriptor.route_id. Sorry I didn't make it clear and I'll going to refine the PR. It should not be possible to replace a trip to work on a different route in a REPLACEMENT trip as it may confuse consumers.

[miklcct]

I do not want to specify the behaviour of missing stops at this moment because it may depend on the client's capability for dynamically adding stops via Stop messages. Theoretically the stop_id must refer to a stop in GTFS static, or a stop added via Stop messages.

[leonardehrenfried]

So this is pretty much a codifcation of what OTP has been supporting for several years. This would of course be very convenient for us but I would like to hear more voices from the community, in particular producers.

I know that HSL (Helsinki) is using this as both a producer and consumer (OTP) for many years.

MBTA has also indicated that they use ADDED as a producer.

@optionsome @jfabi @sam-hickey-ibigroup

[skinkie]

So this is pretty much a codifcation of what OTP has been supporting for several years. This would of course be very convenient for us but I would like to hear more voices from the community, in particular producers.

Producing it for over 12 years too.

[optionsome]

I know that HSL (Helsinki) is using this as both a producer and consumer (OTP) for many years.

HSL doesn't produce or consume ADDED or REPLACEMENT updates currently, if that was what you were referring to.

[leonardehrenfried]

What happens when you ADD a trip and then CANCEL it again? Should it be become invisible in the system (DELETED?) or show up as a CANCELLED?

[miklcct]

That's a good question. I still need to think about how things will work.

My producer implementation cancels an added trip using TripUpdate.schedule_relationship = ADDED with all StopTimeUpdate having a SKIPPED relationship.

The questions are that:

1. What if a later version of the full dataset real time feed doesn't contain the ADDED feed? (My intention is that it no longer exists and should be considered as DELETED. A GTFS-RT full dataset should only be applied into the original static data.)
2. How do I cancel an ADDED trip? I think that a TripUpdate with a trip id not found in static GTFS, schedule_relationship = CANCELED and the original planned stops marked with StopTimeUpdate.schedule_relationship = 'SKIPPED' makes it clear that the trip was added then cancelled. (Use DELETED instead of CANCELED to hide it from the board)

[skinkie]

As we are considering FULL_DATASET shouldn't that just replace?

[leonardehrenfried]

Actually, @skinkie is right. If you use FULL_DATASET then the moment you fetch the new version of the RT feed the old ADDED trip will completely vanish and it neither exists as DELETED nor CANCELLED. It's like it never existed.

However, once there is movement towards specifying INCREMENTAL we will have to revisit this.

[leonardehrenfried]

Does anyone know how Google and Apple handle ADDED?

@bdferris-v2

@eliasmbd I don't know who the relevant person from Apple would be. Could you tag them?

[skinkie]

@leonardehrenfried at Google the thing was limited to stop sequences previously observed. Hence if the ADDED trip was an instance of a stop sequence that is part of the database, it could be processed. I don't know if it is already capable of processing a partial instance of a stop sequence.

https://support.google.com/transitpartners/answer/10106497?hl=en#zippy=%2Cadd-with-tripupdates

[miklcct]

I don't know. I can try to do some research on this, but it's the uncertainty that makes me uncomfortable retroactively adding new required semantics for this trip type.

This is why I think it would be a good idea to start to introduce something like 'compliant with GTFS-RT version' field, so we can proactively distinguish the past and the future.

There is already a FeedHeader.gtfs_realtime_version field in the feed (which is currently 2.0), but it isn't used anywhere in the spec.

If we want to move forward in terms of versioning, I'll need it to be 2.1 to guarantee these behaviour specified in my proposal (then it will need to be bumped to 2.2 if some other underspecified field, such as the behaviour of incremental feeds, are formally specified).

A producer who are aware of this will then have to set the value in the feed to 2.1, while new consumers who are aware of this feature can retain guesswork if the feed version is not specified as 2.1 or above.

[miklcct]

Apologies, forgot to respond to the question about REPLACEMENT. I know of significantly fewer feeds that are still using the REPLACEMENT enum. Maybe 3 depending on how you do the math? Honolulu https://thebus.org/ is one it seems? Given it's been explicitly deprecated for so long and the limited existing usage, I think there is more of a case to be made for keeping the existing tag value.

According to this discussion, Translink was still using REPLACEMENT trips as of 2019.

[bdferris-v2]

I've got lots of thoughts on versioning but the tl;dr is that it's not something I'd support for making frequent backwards-compatible-breaking-for-producers changes to the spec. Which is to say, if we are ever going to bump the major version number for GTFS-Realtime, we'd want to bundle a number of changes together, not just this one.

So put another way, if you want to make progress on this proposal in 2025, I don't think versioning is going to be the quickest path.

[sam-hickey-arcadis]

One additional general comment: only some of the new fields are marked as experimental. Was this intended? It seems all new fields should be noted as experimental.

[sam-hickey-arcadis]

This route_id must be in the GTFS feed? If yes, that should be stated here.

[sam-hickey-arcadis]

For ADDED and REPLACEMENT trips, do stop_time_updates need to be provided for all stops at all times when the trip is in the feed, even when those stop_time_updates refer to times in the past? If yes, it would be good to state that to avoid ambiguity.

[miklcct]

Yes, the text already says that stop_time_updates must be provided for all stops in the added or replacement trip.

[sam-hickey-arcadis]

Agreed. But the first line says, “Updates to StopTimes for the trip (both future, i.e., predictions, and in some cases, past ones, i.e., those that already happened).” The “in some cases” implies past times do not need to be provided, and in practice many producers do not publish past stop times. If all times (including those in the past) need to be always provided for ADDED and REPLACEMENT trips, then I would suggest stating that explicitly.

Another thing to consider from https://github.com/google/transit/blob/master/gtfs-realtime/spec/en/trip-updates.md:

You are allowed, but not required, to drop past stop times. Producers should not drop a past StopTimeUpdate if it refers to a stop with a scheduled arrival time in the future for the given trip (i.e. the vehicle has passed the stop ahead of schedule), as otherwise it will be concluded that there is no update for this stop.

And from https://github.com/google/transit/blob/master/gtfs-realtime/spec/en/reference.md#message-stoptimeupdate:

Updates can be supplied for both past and future events. The producer is allowed, although not required, to drop past events.

It makes sense that all stops with future times on ADDED and REPLACEMENT trips must be published as there is no GTFS trip to link back to, but is there a reason ADDED and REPLACEMENT trips need to include times in the past?

[miklcct]

I've got lots of thoughts on versioning but the tl;dr is that it's not something I'd support for making frequent backwards-compatible-breaking-for-producers changes to the spec. Which is to say, if we are ever going to bump the major version number for GTFS-Realtime, we'd want to bundle a number of changes together, not just this one.

So put another way, if you want to make progress on this proposal in 2025, I don't think versioning is going to be the quickest path.

The original behaviour of ADDED trips was never specified. No one actually knew what it meant. How is it a breaking change if there wasn't even a behaviour to break?

If you relied on some unspecified behaviour for your systems to work, it is your problem.

[mads14]

The original behaviour of ADDED trips was never specified. No one actually knew what it meant. How is it a breaking change if there wasn't even a behaviour to break?

What I've often seen is more like option 1 in the https://gtfs.org/documentation/realtime/examples/migration-duplicated/ where the ADDED trips tripId refers to the static GTFS trip id that is being copied. This would now be strictly disallowed because this proposal states "In this proposal, TripUpdate.schedule_relationship = ADDED should be used to add trips which do not duplicate an existing trip" and "trip_id in the TripDescriptor for added trips must be completely new (not found in GTFS static)"

[bdferris-v2]

I promised I'd come back with a few more data points on my end.

1. While it's true that the behavior of ADDED was never formalized in the spec, Google has documented their supported interpretation for quite a while. Google's interpretation mostly matches what would eventually become DUPLICATED.
2. In practice, I see ~80 feeds that are actively using ADDED in the duplicate-an-existing trip sense, as they are specifying trip ids that match existing trips in the static feed.
3. I will note that I also see feeds that use ADDED for new, unknown trips as well, though it's tricky for me to tell how many of them would pass validation under the changes proposed here.

So I come back to my original point. Though ADDED was never formalized, there are a significant number of existing feeds that are using the enum in a way that would become invalid with the proposed change. Thus, I consider this a backwards-compatible-breaking change and I'm still -1 on the proposal as currently written.

But as I said, I'm supportive of a well-documented path for feeds that want to specify new trips. I think that's best achieved through a new ScheduleRelationship enum tag.

[sam-hickey-arcadis]

Agreed with @bdferris-v2 that even though ADDED was not previously fully defined, the changes proposed here would break existing feeds. I am also in favor of changing this proposal to use a new ScheduleRelationship value.

[miklcct]

The voting period is now over.

The result was 5 yes and 1 no.

According to the Amendment process, for this vote, unanimous consensus is NOT reached, hence this proposal is not accepted.

As the advocate my intention is to continue to work on the proposal.

[eliasmbd]

The voting period is now over.

The result was 5 yes and 1 no.

According to the Amendment process, for this vote, unanimous consensus is NOT reached, hence this proposal is not accepted.

As the advocate my intention is to continue to work on the proposal.

Thank you for your continued advocacy and for keeping us informed of the results. Wishing you the best in your ongoing efforts.

[miklcct]

Unfortunately my proposal isn't workable because there are existing feeds and implementations where TripUpdate.schedule_relationship = ADDED means something incompatible with my proposal, which will result in breaking those feeds and implementations.

As a result, I will work on a new version where the use of TripUpdate.schedule_relationship = ADDED will be completely deprecated. I will use TripUpdate.schedule_relationship = NEW as the enum value to specify the behavior for a completely new trip not found in GTFS static, and mark it as an experimental field. I will also write a transition guide to migrate to NEW trips as well, similar to the one written for DUPLICATED trips.

For REPLACEMENT, I plan to leave what it is now, as the enum has been removed from the spec for more than a decade already, and I am not aware of any residual use of that value in live feeds / other implementations which are incompatible with the proposed (i.e. OpenTripPlanner) implementation.

[skinkie]

I would be more in favor of a version bump than adding a new enumeration.

[miklcct]

I have updated the PR and would like some discussion. I aim to present this for voting (with a NEW enum) in early February.

[sven4all]

I don't like this approach, there for I think it's good to seriously consider a version bump where the 'ADDED' enumeration is completely removed as @skinkie suggested.

[miklcct]

I'm sorry, but as Google mentioned, we can't break backward compatibility.

A decade ago, when REPLACEMENT was removed from the spec, there was a version of .proto file without that value published, and it horribly broke consumers on existing feeds when they were compiled with an updated .proto . Therefore the value has been put back and marked as deprecated.

Therefore the value ADDED has to remain even if we don't want to use it anymore, and those consumers who are using ADDED for the purpose I am migrating to NEW (like OpenTripPlanner) can treat it the same as NEW instead.

[skinkie]

We don't break backwards compatibility when bumping a version.

[tzujenchanmbd]

@sven4all @skinkie
May I confirm what you expect for versioning here? (I saw option A in the beginning of this conversation, just to double check...)

A. Bump version number, add "NEW" and completely deprecate "ADDED"
B. Bump version number, add "NEW", "ADDED" coexist (with transition guide)
C. Bump version number, no "NEW" but make changes on "ADDED"

Perhaps some of these look acceptable or ideal for you?

[skinkie]

I am preferring C. I absolutely don't want a situation which is suggested by B where a bumped version exists, and data ends up in both NEW and ADDED under that higher version.

[bdferris-v2]

Let me be up front that if we are considering a version bump to GTFS-Realtime (either major or minor), then I don't think we'd do it for just this proposal. We've never done a version bump of GTFS-Realtime, so this would need a broader discussion of whether we'd actually do it, what the process looks like, expectations around support for old and new versions, and if we come to terms with all that, then an extended phase where we discuss any other breaking-changes that we'd want to roll into the new version (think something akin to the GBFS process).

Which is to say, I don't want you to be surprised if/when I vote -1 on any single proposal here with a version bump.

[miklcct]

The GTFS-realtime version is now 2.0, where it was 1.0 in the past, so it has been done already.

[bdferris-v2]

True, but my concern still stands.

[miklcct]

Also, when REPLACEMENT was removed a decade ago (because it was never fully specified, like ADDED right now), breaking changes to the .proto was actually released into the wild as well.

[gcamp]

I wouldn't consider 2.0 a real breaking change. In practice, it only formalized that some states were invalid. In the past, those invalid state were valid per the specification but no one really considered those state valid. #64

In other words, I don't think anyone has special code handling gtfs-rt 1.0 and 2.0.

[miklcct]

I need as many voices as possible now, no later than 2025-01-31, and I will present an updated proposal for voting again soon afterwards.

[skinkie]

@eliasmbd can you gather your members?

[eliasmbd]

In light of the feedback here and the complexities surrounding versioning, we believe this proposal would benefit from additional discussion and refinement as the GTFS Realtime versioning process has yet to be formalized despite its historical presence in the specification.

To that end, we propose to open the conversation to a wider audience with a separate issue dedicated to discussing GTFS Realtime versioning. This will provide a structured space to address the broader implications of version changes.

We’d like to remind everyone of some key governance aspects:

• While the governance requires a minimum of 7 days discussion, they should continue for as long as the Advocate deems necessary to address concerns and refine the proposal.
• Thorough discussions prevent misunderstandings and allow for all opinions from the community to be shared.
• Reaching consensus requires accommodating diverse viewpoints to ensure the proposal benefits the global community of producers and consumers.

In some cases, moving slower and dedicating time to fully address concerns can lead to quicker and more effective results in the long run, as it reduces the likelihood of revisiting unresolved issues.

We encourage the community to continue engaging constructively, as this collaboration is essential to maintaining and evolving GTFS Realtime in a way that serves everyone.

Thank you again for your commitment and input.