Runbook: changing what appears in a Qualtrics survey

Six automations on Posit Connect write to Qualtrics survey definitions every day — they change the questions themselves, not the response data. This page is for the person who needs a dropdown to say something different tomorrow morning. It tells you what to edit, how long it takes, and what you cannot take back.

It deliberately does not explain how the jobs work. Each .qmd opens with a prose header covering its own mechanism, and those are the authority — this page links to them rather than repeating them.

Read this first

Adding a row to the FY27 Official Site List tab is a live survey change.

That tab is the only gate between a spreadsheet edit and six live surveys. When update_site_choices next runs, a new row there can:

add the site as a choice on the site question in seven surveys, and
create a brand new subsite question in the six surveys that own a site block, if the site also has a column on the subsite tab.

Nobody reviews this. There is no staging copy. The run happens at 06:30 daily and the questions are live to respondents as soon as the survey republishes.

Two guard rails exist and it is worth knowing their limits. A subsite column is ignored unless its site is on the Official Site List — that is the gate, and it is why the list matters so much. And max_new_questions_per_survey (10) aborts the run rather than spraying questions if the sheet changes shape. Neither protects you from adding one wrong row correctly.

If you are not sure a site is real yet, leave it off the Official Site List. A column can sit on the subsite tab indefinitely doing nothing.

Three rules that apply to everything here

1. Choices are additive. You cannot cleanly withdraw one.

No job here ever deletes or renames a choice. Once “Ava High School” is a choice in a live survey, it is there permanently. The reason is that a choice id is referenced by display logic, the PF survey flow, the subsite cascade, and every response already recorded against it — deleting it breaks all of those silently, with nothing erroring.

The available remedies are worse than they sound:

Retiring a site: it stays in the dropdown. qualtrics_checks reports it as a survey-only entry so a human can hide it deliberately.
A coach or facilitator who leaves: their choice is set to display logic False, so it is hidden, not removed.

2. A whole question can be deleted — but only while it holds no responses.

This is the one real undo, and it only applies to subsite questions created by update_site_choices. delete_question() at the bottom of that file is the documented escape hatch. Use it the same day; once a single respondent answers, you are back to rule 1.

3. Export tags are minted automatically and are hard to change later.

When a new subsite question is created, its export tag is derived from the site name — CA_Lighthouse Community Public Schools becomes ca_lighthouse. That tag becomes the column name in every data export, so every minted tag is printed in the run log for you to check.

If you want a different one, set it in question_tag_overrides in update_site_choices.qmd before the first run creates the question. Renaming a tag afterwards orphans the export column and every response already filed under it. FL_FAMU DRS → fl_famu is the existing example.

Also worth knowing: new subsite questions are always created as radio buttons (SAVR). That is fine for a handful of schools and unusable for a long list — if a site has dozens of subsites, switch the question to a dropdown by hand in Qualtrics afterwards. The automation will keep topping up its choices either way.

I need to…

…add a new site

Edit: the FY27 Official Site List tab of the FY27 Master Survey Input sheet, column Partner Name in Survey. Use the exact PREFIX_Name convention — the name you type is the choice text respondents see, and it is permanent.

Appears: next run of update_site_choices, 06:30 daily.

Changes: the site question in all seven FY27 surveys (Participant Feedback, Educator Survey, IPG, Student Work, Mid/End of Year Lead, Enabling Conditions, Teaching Lab Student). If the site also has a column on the subsite tab, a new subsite question is created in the six surveys that own a site block.

Cannot be undone: the site choice. Check the spelling before you save. A newly created question can be deleted while it has no responses.

Do not add FY27 sites to the old shared reference survey (LS_4YF9JoK4AVZdADY). It is frozen and only FY26 surveys still use it.

…add schools or districts under an existing site

Edit: the FY27 School/District Selection for Sites tab, in the column named for that site.

Appears: next run of update_site_choices, 06:30 daily.

Changes: the subsite question for that site, in the six surveys with a site block. New values are added; nothing existing is touched.

If nothing happens, the usual cause is that the site is not on the Official Site List — the gate. The run log prints every ignored column under “Subsite columns absent from the Official Site List”.

Two sites are deliberately never driven and this is not a bug: DC_DCPS resolves to two questions (9 clusters and 117 schools) and the one-column-per- site sheet cannot say which you meant, so it is reported and skipped. The tn_acceleration2 cascade hangs off the region question rather than the site question, so it never claims a site. Edit both by hand in Qualtrics.

NYC districts (district9, district11, district75, …) are text-entry questions filled by JavaScript, not choice lists. Adding subsites for them on the sheet does nothing by design.

…remove a site

You cannot, and you should not try. See rule 1. Take the site off the Official Site List — it stops receiving new subsites, and qualtrics_checks will list it as a survey-only entry. Hiding the choice in the live survey is a manual Qualtrics edit, and a deliberate one.

Five sites came off the FY27 list this way and are awaiting confirmation that they are genuine exits: ar_osceola, ct_hamden, ma_westspring, tx_elpaso, tx_la_joya. See OPEN_WORK.md.

…make a course appear in Participant Feedback

Edit: nothing here — the course list is not hand-maintained. It comes from the Monday PL logistics boards, collated daily into tl-monday/pl_today_courses_fy27.rds by update_fac_course_list_daily_fy27 (06:00), covering courses within roughly a week either side of today.

Appears: update_course_choices at 07:00, writing per-site display logic on the course question (QID221) in FY27 Participant Feedback (SV_cBaoLkjhoXkOG4S).

So if a course is missing, fix the Monday logistics board — the session date, the L&R Course Name, or the L&R name site. The survey is downstream of that.

Ask: L&R for the board content.

…fix who shows up as facilitator on Participant Feedback

Edit: the Monday PL logistics boards again. The facilitators for sessions near today are collated into tl-monday/pl_today_facilitators_fy27.rds daily.

Appears: update_facilitator_choices at 07:00, writing per-site display logic on both PF facilitator questions — QID224 (tag facilitator) and QID109 (tag coach_2) in SV_cBaoLkjhoXkOG4S.

Note the trap: QID109’s export tag is literally coach_2, but it is maintained by the facilitator job, not the coach job. See the section below.

…add or hide a coach

Edit: the Monday facilitator board (tl-monday/monday_facilitators.parquet) or the employee board (tl_employees.parquet). Someone counts as an active coach if their Coach Onboarding is Full Time, Done or In Progress, or they are an FTE, or their Able to Support mentions Coaching.

Appears: update_coach_selection at 04:54.

Changes: the coach question in FY27 IPG (SV_0DGUXmHmlnTOZim), FY27 Student Work (SV_0H7xUSb8qixVCDk), and Coach-Teacher Recorded Conversations SY25-26 (SV_b9pikOzhbXvOapw).

Removing someone hides them (display logic False); it never deletes the choice, so historical responses stay intact.

…add a teacher or coachee

Edit: the coach roster on Monday, read via TeachingLab::get_coach_roster(year = "26_27").

Appears: update_coachee_selection_27 at 03:54.

Changes: the ts_* questions in FY27 IPG, FY27 Student Work, Coaching Cycle Student Outcomes (SV_bOSNHJtpzbvUyFg), and Coach-Teacher Recorded Conversations. If a site in the roster has no matching ts_ question, one is created with a single “Other” choice.

…change the Podsie / Coteach branching in Participant Feedback

Edit: the Product NYC column on the FY27 Monday PL boards.

Appears: update_pf_flow at 07:00, rewriting the PF survey flow (not a question) for sites with a session within three days.

Caveat: the FY27 boards do not carry a Product NYC column yet. Until they do, the job warns and leaves both flow branches untouched rather than failing.

Facilitator vs coach: these are two different jobs

update_facilitator_choices.qmd and update_coach_selection.qmd look like duplicates. They are not, and editing the wrong one is the most likely mistake on this page. The confusion is real rather than cosmetic: the same person is frequently both a facilitator and a coach, so you cannot tell which job owns a name by looking at the name.

Tell them apart by which survey and what kind of change:

	`update_facilitator_choices`	`update_coach_selection`
Surveys	FY27 Participant Feedback only	FY27 IPG, FY27 Student Work, Coach-Teacher Recorded (SY25-26)
Questions	`QID224` (`facilitator`), `QID109` (`coach_2`)	the question tagged `coach` in each survey
Source	S3 `pl_today_facilitators_fy27.rds`, refreshed daily from the Monday PL logistics boards by a separate Connect job	Monday facilitator board + full-time employees
What it writes	per-site display logic — who is shown to a participant at this site	the choice list — who exists as a coach at all
Question asked	“who ran your session today?”	“who is your coach?”
Runs	07:00	04:54

The short version: Participant Feedback is the facilitator job’s territory, and nothing else is. If the survey you care about is PF, you want update_facilitator_choices. If it is IPG or Student Work, you want update_coach_selection.

Two traps worth stating plainly:

QID109 is tagged coach_2 but belongs to the facilitator job. The tag is a historical artefact. Do not go looking for it in update_coach_selection.
update_coach_selection used to handle PF too. That half was split out into update_facilitator_choices, and the original file’s header went stale for a while claiming otherwise. Both files now cross-reference each other; trust the headers as of 2026-08-07, not older copies.

FY27 state of the coach job

Until 2026-08-07 update_coach_selection was still pointed at the FY26 IPG and Student Work surveys — it had been quietly maintaining last year’s coach lists while the FY27 surveys received nothing. It now targets FY27, and resolves its target question by export tag (coach) rather than a hardcoded QID, so the same silent failure cannot recur: a wrong survey id now stops the run instead of finding a plausible question.

Coach-Teacher Recorded Conversations is deliberately still the SY25-26 survey (SV_b9pikOzhbXvOapw). No FY27 copy exists anywhere in the account and that one is still collecting. Whether FY27 should have its own is an open question for L&R — see OPEN_WORK.md.

The six jobs at a glance

Job	Reads	Writes to	Runs
`update_site_choices`	FY27 Master Survey Input sheet: Official Site List + School/District tabs	site + subsite questions in 7 surveys; creates subsite questions; publishes the registry parquet	06:30
`update_course_choices`	`tl-monday/pl_today_courses_fy27.rds`	course question `QID221`, FY27 PF	07:00
`update_facilitator_choices`	`tl-monday/pl_today_facilitators_fy27.rds`	`QID224` + `QID109`, FY27 PF	07:00
`update_coach_selection`	Monday facilitator board + employees	`coach` question in FY27 IPG, FY27 SW, CTR SY25-26	04:54
`update_coachee_selection_27`	coach roster (`get_coach_roster("26_27")`)	`ts_*` questions in 4 surveys	03:54
`update_pf_flow`	`tl-monday/all_pl_boards_fy27.parquet`	FY27 PF survey flow (Podsie/Coteach)	07:00

qualtrics_checks and update_fac_course_list_daily_fy27 also touch survey definitions but only read them — they cannot change a survey.

When something goes wrong

Start with the run log. Every job renders on Connect and the log is the record of what it did and why. All of them are chatty on purpose: ignored subsite columns, minted export tags, coaches added and hidden, and sites needing a question are all printed.

Then decide whose problem it is:

The survey shows the wrong thing, and the source is right → the automation. Check whether the job’s last render succeeded, and whether the deployed bundle is older than the committed source. Live deploy state is in ARCHITECTURE.md.
The survey matches the source, and the source is wrong → L&R. Sheet tabs, Monday board columns, session dates and site names are all theirs.
Something is in the survey that should not be → read rule 1 before doing anything. If it is a question created today with no responses, delete_question() is the undo. Otherwise it is a deliberate manual hide, not a deletion.

Do not fix things by hand in Qualtrics without also fixing the source. These jobs are idempotent and run daily — a manual edit that contradicts the sheet or the board will be quietly re-applied tomorrow morning. The exceptions, which the automation genuinely cannot express, are: switching a new subsite question from radio buttons to a dropdown, hiding a retired site’s choice, and the two never-driven questions (DC_DCPS, tn_acceleration2).

Ask: L&R for anything upstream — site lists, subsites, rosters, board columns. The data team (duncan.gates@teachinglab.org) for the automations themselves; update_coach_selection, update_facilitator_choices and update_coachee_selection_27 already email there on every change.