Troubleshooting Guide
The Training Village is a multi-component system where hardware, software, and behavior intersect. Minor miscalibrations or component failures can lead to suboptimal performance. Typically, small issues may arise during the first few days of deployment as the animals habituate and the sensors are fine-tuned.
When a system anomaly is detected, an automated alert is instantly dispatched via Telegram. It is crucial to respond promptly by logging in remotely to investigate the root cause of the error, reviewing the corridor and operant box video recordings. For example: a door that does not close properly or a lighting failure can produce several types of errors: incorrect RFID detection, two animals simultaneously entering the operant box, the system registering an animal as present when the box is empty, or the system failing to detect an animal that is actually inside.
Where to Look First
The events log (
DATA→EVENTS): every action, refusal and error is written there. Double-click a row to see its full description and, for errors, the Python traceback with the line of code that failed.The videos: select an event and click
VIDEOto see the corridor at that moment, or open the session video fromDATA→SESSIONS_SUMMARY.The terminal where
villagewas launched, which also shows the messages printed by your own code.The logs of the synchronization and of the video deletion (
rsync_logs/anddata_removal_logs/, indata/<SYSTEM_NAME>/).
Illumination & Tracking Issues
Many gating or tracking failures are not caused by software bugs, but by physical lighting conditions.
Symptom: Double Entries or Wrong RFID detections
If you notice that two animals are entering the operant box simultaneously, or that a gate closes while an animal is still in the corridor, the cause is usually optical reflections.
The Cause: The computer vision algorithm relies on a clean binarized pixel count. If there are glare reflections on the clear acrylic corridor ceiling, the camera can become “blind” to those reflection zones. For example, if a second animal is standing directly under a heavy glare reflection, its pixels will not be counted. The system will assume the corridor is empty and mistakenly open the gate for an RFID-tagged animal, leading to double entries or wrong RFID detections.
The Solution: Adjust the physical positioning or angles of your external room lights. Ensure the corridor is shielded from direct overhead glare. While the camera automatically adjusts its exposure between day (
VISIBLE) and night (IR) modes to keep images consistent, it cannot filter out localized mirror-like reflections on plastic surfaces.
Symptom: Broken Video-Tracking inside the Operant Box
If the advanced center-of-mass tracking (findContours) is dropping or jumping erratically inside the behavioral arena, inspect the box illumination profile.
The Cause: The operant box tracking requires highly uniform illumination (either visible or IR, depending on your paradigm).
The Solution: Ensure the LED strips inside the box are clean and fully functional. Never open the operant box door while a behavioral session is actively running, as the sudden influx of external room light will completely disrupt the binarization thresholds and crash the real-time tracking loop.
Camera Issues (Connection Error or Malfunctioning)
Verify Physical Connections: Ensure both camera ribbon cables are firmly seated in the Raspberry Pi 5 MIPI CSI slots and that the white locking clips are fully engaged.
Check Camera Indices: Open the application GUI, navigate to
SETTINGS→CAMERA SETTINGS, and verify that your camera indices are properly assigned (typically one is0and the other is1).CAM_CORRIDOR_INDEX:0or1CAM_BOX_INDEX:0or1
Reboot: If you swapped indices or re-seated the cables, restart the system to allow the OS to re-initialize the video devices.
Controller Issues (Bpod or Arduino Connection Error)
Verify Physical Port: Ensure your behavioral controller is plugged into the bottom USB 3.0 port (the bottom blue port closest to the Raspberry Pi’s onboard Ethernet connector).
Verify App Settings: Navigate to
SETTINGS→CONTROLLER SETTINGSin the GUI. Ensure your specific controller type is selected and the device path is set exactly to/dev/controller.Check System Devices: Open a terminal and check if Linux detects the hardware as a raw serial interface by running:
ls /dev/ttyACM* /dev/ttyUSB*
Isolate Hardware: If no device appears in step 3, try replacing the USB cable. Use a shorter, high-quality, shielded USB cable to prevent electromagnetic interference.
Verify Symbolic Link: Check if the automated udev rule successfully generated your symlink by running:
ls -l /dev/controller
Fix Missing Symlink: If the raw device (
ttyACM*orttyUSB*) appears but/dev/controlleris missing, inspect the system logs by typingdmesgto find the exact port configuration, then review your rule file at/etc/udev/rules.d/99-usb.rules.
Understanding Symlinks, udev Rules & /dev/controller
A symlink (short for symbolic link) is a software shortcut in Linux that transparently redirects one file path to another.
Why it is necessary: When you connect a serial device (like a Bpod or Arduino) to the Raspberry Pi, Linux dynamically assigns it a generic node path like
/dev/ttyACM0or/dev/ttyUSB0. However, if the device is unplugged, if the Pi reboots, or if another USB peripheral is connected, Linux might randomly reassign that hardware to/dev/ttyACM1. If the Training Village software were hardcoded to look for a static name likettyACM0, the connection would instantly break.The Automated Solution (
udevrules): To prevent this, Linux uses a background device manager calledudev. The Training Village comes pre-configured with a custom rule file (located at/etc/udev/rules.d/99-usb.rules) containing the following instruction:SUBSYSTEM=="tty", KERNELS=="1-1:1.0", SYMLINK+="controller"How this udev rule works:
SUBSYSTEM=="tty": Filters the incoming hardware events and tells Linux to only trigger this rule if the connected device is a serial/terminal interface (like a USB microcontroller).KERNELS=="1-1:1.0": instead of looking at what device is plugged in, it looks at where it is plugged in.1-1:1.0corresponds to the exact physical address of the bottom USB port adjacent to the Raspberry Pi 5 Ethernet jack.SYMLINK+="controller": If a serial device matches the criteria above, udev automatically creates a permanent, unchangeable symbolic link shortcut named /dev/controller pointing straight to it.
Thanks to this rule, no matter what microcontroller you plug into that specific bottom port, and regardless of the generic name Linux gives it internally, the Training Village software can always find your behavioral controller seamlessly.
Peripheral Issues (Sensors, Motors, Lights, or Satellite Boards)
Verify Software Activation: Open the GUI and ensure the specific satellite board you are trying to communicate with is enabled: toggle
USE_CORRIDOR(top ofSETTINGS→CORRIDOR SETTINGS) orUSE_BOX_BOARD(top ofSETTINGS→BOX SETTINGS) toONdepending on your active setup.Verify Satellite Connections: Check that the failing component (servo, LED strip, RFID antenna) is securely connected to the correct ports on the Corridor Board or Box Board with the correct wiring polarity.
Check Hub-to-Satellite Cables: Each satellite board is linked by two cables — an Ethernet cable carrying data, and a separate power cable (DC barrel jack on both ends) carrying 5V. Ensure both are firmly plugged into their respective dedicated ports on the Main HAT.
Check HAT Seating: Ensure the Main HAT is fully pressed down and evenly seated onto the Raspberry Pi 5 GPIO header extension pins.
Test Power Lines: Verify that the Main HAT is receiving a stable 5V (3A) from the 5V power supply. A drop in voltage can cause I2C chips, scales, or servos to drop off the bus randomly.
Restart Village After Reconnecting a Motor: Physically unplugging and replugging a servo can momentarily glitch the power rail feeding its PWM driver chip (PCA9685), resetting the chip’s internal PWM frequency configuration. The Raspberry Pi’s I2C connection to the chip keeps working (no error is shown), but the servo stops responding correctly because the pulse timing it’s now receiving no longer matches what it expects. That frequency is only configured once, when Village starts — if a motor stops working after being disconnected and reconnected, restart Village to reinitialize it, rather than assuming the servo itself is broken.
An Animal Never Enters the Box
When an animal is detected by the RFID reader but not let in, the reason is written in the events log. Check, in this order:
The RFID reader is ON (
MONITOR→CORRIDOR), and the tag is being read: every reading writes “Tag detected: …” in the events.The subject is registered with that tag. Otherwise the events show “Subject with tag: … not found” (see Subject Management).
The subject is active now. “Subject not active” means its
activefield is OFF or the current hour is outside its schedule.Its refractory period is over. “Subject not allowed to enter until …” shows its
next_session_time, which is the end of its last session plusrefractory_period.The corridor looks right to the camera. “Detection in area1: …”, “Detection in area2: …” or “Large detection in area3: …” mean that, for the camera, there is something else in the corridor or two animals in area 3. Check the lighting, dirt and the thresholds (see Animal Detection & Tracking).
Only one tag was read. “Multiple tags detected in the last seconds” means another animal was detected less than
TIME_BETWEEN_DETECTIONSseconds before.
The corridor camera must also be giving an image: without it, no animal is let in.
Errors at Startup or When Launching a Task
“System running in debug mode” when Village starts: a device could not be started. The message lists which ones; see the sections above for cameras, the controller and the peripherals.
“Couldn’t import
” in the events: a Python file of yourcodefolder has an error. Double-click the row to see the traceback. Everything in that file (tasks, triggers, plots…) is ignored until it is fixed. “No tasks could be imported” and “Training protocol not found” usually come from the same cause.“Error running task: … not found”: the subject’s
next_tasknames a task that doesn’t exist (renamed or deleted). Fix the training protocol, or the subject’snext_settingsin theSUBJECTSscreen.“WATER CALIBRATION PROBLEM” / “SOUND CALIBRATION PROBLEM” when a task starts: the task asked for a volume or a sound level that is not calibrated, or out of the calibrated range. Calibrate it (see Water Calibration and Sound Calibration), or change the requested value.
“Error running task …” while the task runs: an exception in the task code. The traceback shows the line.
The Animal Doesn’t Leave (Scale and Exit Detection)
The system decides that the animal has gone back home by combining the corridor camera
and the corridor scale. With door 2 open, it waits until areas 3 and 4 are empty, the
animal is in area 2, and the scale gives a stable weight between
MIN_WEIGHT_THRESHOLD and MAX_WEIGHT_THRESHOLD (10 to 40 g by default, in
SETTINGS → CORRIDOR SETTINGS). Only then it closes door 2, opens door 1 and writes
“Subject back home: … g”.
If the scale is not working, is badly calibrated, or the thresholds don’t fit your
animals, the system keeps waiting. Once the task has reached its maximum_duration,
if the animal is still not detected back home one hour later, it sends “The subject
has been in the box for too long.” (repeated every hour).
Check the scale from
MONITOR→CORRIDOR:TARE SCALEwith nothing on it, thenGET WEIGHTwith a known weight. If it is wrong,CALIBRATE SCALEagain.Check the thresholds against the weight of your animals. For rats, or for heavier mice,
MAX_WEIGHT_THRESHOLDmust be raised.If an animal is stuck, help it out and use
CHANGE STATE→ “All subjects are back home, go to WAIT state” (see the Toolbar).
After a Power Cut
The Raspberry Pi starts again by itself when the power comes back, but the Training
Village does not: it has to be launched again (village, see
Launching the TV Application). Until then, no animal can
enter the box, and the healthchecks.io heartbeat stays silent, so the team gets the
heartbeat alarm.
Before launching it, save the interrupted session if you need it. While a session runs, its events are written as they happen to
data/sessions/session.csv, and the session files are only created when the session ends. That file is emptied when the Training Village starts, so copy it somewhere else first. The interrupted session is not added to the subject’s data, and its settings are not updated.Launch the Training Village. It starts in
WAIT, without moving the doors.Check, with the box camera, whether an animal was left inside. If so, open door 2 from
MONITOR→CORRIDORso that it can leave, and when it is back home leave door 1 open and door 2 closed.Check that the cameras, the RFID reader and the Telegram bot work again, as after any start.
No Sound
The DAC is still used by the operating system. Its device profile must be set to Off (see Sound).
The sound device is not selected:
SETTINGS→SOUND SETTINGS→USE_SOUNDCARDON, and the rightSOUND_DEVICE.The sound card mutes itself after a silence: disable its auto-mute (see the manual installation).
“Error in sound device” in the events: double-click it to see the cause.
Screen or Touchscreen Issues
Nothing is shown on the screen of the box: the system must be set to always detect two screens (
vc4.force_hotplug=3in/boot/firmware/cmdline.txt), and the second screen must be connected to the second micro-HDMI port (see Screen & Touchscreen).Touches are not registered: check
TOUCHSCREEN_DEVICE(the exact name of the device) andTOUCH_RESOLUTION.Touches move the mouse pointer of the desktop: the udev rule that tells the desktop to ignore the touchscreen is missing (see Screen & Touchscreen).
The RFID Reader Doesn’t Read
Check the wiring of the reader (the connector order of the ID-20LA, see the Assembly Guide) and that the Corridor Board is powered.
The serial port must be enabled (
enable_uart=1, see the manual installation; already done in the disk image).Check the chip: hold the animal over the reader. A chip that is too deep or badly oriented may only be read from some positions.
Remote Access or Telegram Bot Not Responding
Remote access is lost but the bot answers: send
/restart_anydeskor/restart_vnc(see Alarm System Overview).The bot doesn’t answer anything: either the Training Village is not running, or the Raspberry Pi has no internet connection. This is the same situation as the heartbeat alarm.
The bot never answered: check
TELEGRAM_TOKENandTELEGRAM_CHAT(see Telegram Bot Integration).
After Updating the Training Village
If, after a git pull, the Training Village fails to start because a Python module
can’t be imported, the new version needs a new dependency. Install it inside the
Python environment:
source ~/.env/bin/activate
cd ~/village
pip install -e . --config-settings editable_mode=compat
Data Synchronization Issues
If you suspect a transfer issue, you can inspect the detailed synchronization logs generated by the system. The log files are stored locally in the following directory:
/home/pi/village_projects/YOUR_PROJECT/data/YOUR_SYSTEM/rsync_logs
If the synchronization fails for 24 hours, the system sends the alarm “No data sync in
the last 24 hours.” to your team’s Telegram group chat (see
Alarm System Overview). Review the logs inside the rsync_logs folder to
identify the specific transfer error, and verify your network connectivity or remote
storage availability.