When Does Normal Activity Become a Security Event? Training AI to Understand Surveillance Video

A security camera can see a person walking through a corridor without any trouble. That is the easy part. The harder question is what the system should actually understand from that movement. Is the person simply passing through, or did they enter an area they should not be in? Have they been standing near an access point long enough for it to matter? Did they leave that bag beside them intentionally, or are they coming back for it? And if they walk behind a parked van and appear again a few seconds later, is it even the same person?

These are very different security situations, even though the camera sees roughly the same things each time ; a person, a doorway, a bag, maybe a vehicle passing through. This is exactly where security and surveillance annotation goes beyond simply drawing boxes around objects. To train a system that understands what is actually happening, the dataset needs to capture several things at once:

• What’s visible in the frame
• Where it is, relative to a defined boundary or zone
• Whether the same person stays identifiable across many frames
• How long an activity continues
• What the person is interacting with
• When the event actually starts, and when it ends

Object annotation teaches a model what’s in the scene. Event annotation teaches it what the scene means. That second part is where most of the difficult decisions begin.

1. Spotting a Person Is Easy. Understanding Them Isn’t.

Most security AI systems start with fairly basic visual understanding ; spotting people, vehicles, bags, doors, gates, fences, equipment, and restricted areas within a frame. With image annotation for security and surveillance, all of this can be marked using bounding boxes, segmentation masks, keypoints, or attributes, giving the model clear and reliable ground truth. A box around a person tells the model, in simple terms, that there is a person there. But a real security system usually needs to answer something more useful: what is this person actually doing, and does it matter?

A person standing near a warehouse entrance is completely normal. The same person standing inside a restricted loading area might not be. Someone holding a bag is unremarkable; a bag left behind after they have walked away can become a genuine security event. Consider how differently the same basic objects can be understood depending on context:

• A person + an entrance – usually nothing worth flagging
• A person + a restricted zone – potentially an intrusion
• A person + a bag, together – normal
• A bag, alone, person gone – potentially abandoned

This is why two annotators can agree perfectly on a person’s bounding box and still disagree completely about what event, if any, is taking place. The object-level annotation can be accurate while the security interpretation behind it is wrong ; and that distinction affects almost every serious security data annotation project from the beginning.

2. Where It Happens Decides What It Means

Before a system can flag an intrusion, it needs to know where that intrusion can actually happen. This is why most surveillance datasets include virtual zones, usually drawn once for a fixed camera and then used to interpret everything the camera tracks afterward:

• Restricted areas and employee-only sections
• Entrances and exits
• Loading zones
• Perimeter boundaries and virtual tripwires
• Other high-risk areas specific to the facility

Drawing a shape around a restricted area sounds simple enough. The real difficulty is deciding exactly when someone has “entered” it. Suppose a worker walks up to the boundary and one foot crosses the line ; does that count? What if they reach across the boundary but keep their body outside it? What if the camera angle makes their bounding box appear to overlap the zone, even though they are physically standing outside it? These are not rare or hypothetical situations. They are exactly the kind of cases that create inconsistent labels when a project has not defined its rules clearly enough.

Because of this, every project needs one clear, agreed-upon crossing rule. A few of the common ones we work with:

• Any part of the body entering the zone counts
• Only the torso crossing the line counts
• The centre point of the person’s box has to enter
• A minimum percentage of the body has to be inside
• The person has to stay inside for a minimum amount of time before it counts

There is no single rule that works everywhere ; a high-risk industrial area may need a very sensitive threshold, while a wide public entrance might need something more flexible. What matters far more than which rule is chosen is that everyone on the team applies the same one.

A zone only becomes useful once everybody agrees on what interacting with it actually means.

3. One Person, One Story: Why Tracking Holds the Whole Event Together

A security event almost never exists inside a single image ; it develops over time, which is exactly why video annotation for security and surveillance matters so much. Picture a person walking through a facility: the system detects them in frame one, then frame two, then frame three. A tracking ID tells the model that all three detections belong to the same person, and that continuous identity allows the system to understand movement instead of treating every frame as a separate scene.

Here’s why that matters in practice. Suppose someone is tracked correctly for 150 frames, then walks behind a parked van and disappears. Three seconds later, they come back into view. If the annotator gives them a brand-new tracking ID after that gap, the individual boxes might still look perfectly correct ; but the event itself has quietly been split into two. Instead of recording one person moving through the scene, the training data now says one person disappeared and a different person appeared. That kind of break can affect:

• Intrusion detection
• Loitering timers
• Entry and exit counts
• The link between a person and their belongings
• Abandoned-object detection

Tracking therefore needs its own set of rules ; how long an ID should remain active during an occlusion, when a track should be considered finished, when a new ID is genuinely needed, how to handle two people crossing paths, and what to do when re-identification is uncertain enough that it should go to a human reviewer instead of being guessed.

A box can be completely correct in every single frame while the track connecting all of those boxes is wrong ; which is exactly why video quality checks cannot be done by looking at isolated frames one at a time.

4. The Clock Is Part of the Label

Some security events are not defined only by what a person does, but by how long they keep doing it. Loitering is the clearest example. A person standing near an entrance for ten seconds is normal. Thirty seconds is probably still fine. Five minutes might be worth a second look ; but even that is not automatically suspicious. Compare these three situations, all involving someone standing in the same place for several minutes:

• A security guard stationed near the entrance for an hour ; expected, not an event
• A delivery driver waiting several minutes for someone to answer the door ; expected, not an event
• An unfamiliar person with no clear reason to be there, standing for the same stretch of time ; worth flagging

The activity can look almost identical on screen while its actual meaning is completely different. This is why “loitering” cannot simply be given to annotators as a single class name. The project needs to define:

• A minimum dwell time before it counts
• How far someone can move around and still count as staying put
• Whether pacing back and forth counts
• Whether stepping away briefly resets the clock
• Whether a short occlusion pauses the timer
• Whether certain roles or zones get different thresholds

Without answers to these questions, annotators end up using their own judgment of what “too long” means ; and once different interpretations enter the dataset, the model starts learning those differences too. In surveillance data, time is not always just another detail worth noting. Sometimes it is the thing that decides the label outright.

5. Where Does an Event Actually Begin?

Once a project moves from labelling objects to labelling complete events, a new difficulty appears: where exactly does the event begin? Take intrusion as an example. A person approaches a boundary, crosses it, stays inside, then leaves ; the sequence sounds obvious enough on paper, but should the start frame be the moment they first touch the boundary, the first frame their body actually crosses it, the frame their centre point enters the zone, or only after a minimum dwell time has been reached?

Abandoned objects are even trickier. A person carries a bag, sets it down, releases it, walks away, and the bag stays behind. Does “abandoned” begin at any of these points?

• The moment the bag touches the floor
• The moment the hand lets go of it
• The moment the person starts walking away
• Once they’ve moved past a certain distance
• Only after the bag has sat there unattended for a set period of time

The right answer depends entirely on how the system is meant to work, but it cannot be left to each annotator to decide individually ; that quickly creates a dataset full of small, hard-to-notice inconsistencies. A strong event definition explains the condition that must be true before the event can start, the exact starting condition, what the “event is happening” state looks like, the ending condition, and whether any context just before or after the event should also be kept.

This is where annotation guidelines stop being simple documentation and start becoming the actual operational definition of what the model is being trained to learn.

6. The Best Negatives Look Almost Like the Real Thing

Every security dataset needs clear positive examples of the events it is trying to detect. It also needs something just as important, if not more so: normal, everyday activity that looks very similar to those events without actually being one. These are often called hard negatives, and they matter because real systems rarely fail on the obvious cases ; they fail on situations that look close enough to confuse them.

Looks like Actually is
Intrusion	A person walking beside a restricted zone, never entering it
Loitering	A delivery driver waiting outside an access-controlled door
Abandoned object	A worker setting a bag down while rearranging tools
Tailgating	Two authorised employees walking through a door together
Suspicious movement	A technician circling the same machine while servicing it

If a training dataset only contains obvious intrusions and completely empty, uneventful footage, the model can perform very well in testing and still flood a real security team with false alerts once it is deployed. A stronger dataset intentionally includes these confusing, near-miss cases ; movement close to a boundary, brief stops, temporary object placement, genuinely authorised access, events that start but never fully happen, repeated but completely legitimate activity, short occlusions, and crowded entrances.

Some of the most valuable negative examples in the entire dataset are the ones that look almost identical to the event you’re actually trying to detect.

7. When the Camera Loses Someone, the Story Can Break Too

Occlusion ; a person briefly disappearing from view ; is often treated as a simple visibility problem. In surveillance annotation, it can cause much more damage than that. Picture someone setting a bag down beside a pillar and starting to walk away, just as a forklift passes between them and the camera. For a few seconds, they disappear from view, and then they appear again. If the track continues correctly through that gap, the whole event stays connected. But if a new identity is assigned instead, several things can quietly break at once:

• Who actually placed the bag
• How long the person has genuinely been away
• Whether they came back
• Whether the object is still linked to them
• Whether the same person went on to cross another zone

Because of this, annotation rules need to clearly separate a short, temporary occlusion from a genuine loss of track, and both of those from a truly new identity. A workable process usually sets a maximum occlusion window, defines the conditions for confidently re-identifying someone, allows an “uncertain” track state instead of forcing a guess, and sends genuinely ambiguous cases to a reviewer rather than resolving them on the spot.

Visibility and identity are two different problems. A person can stop being visible for a while without becoming a different person in the eyes of the dataset.

8. Not Every Frame Deserves a Yes-or-No Answer

Production surveillance footage is rarely clean. Teams regularly work with footage affected by:

• Low light and infrared footage
• Heavy video compression and motion blur
• Headlight glare and reflections
• Rain and fisheye distortion
• Camera vibration, distant subjects, and partial silhouettes

The problem starts when a guideline forces a confident yes-or-no answer in situations where the footage simply does not provide enough information. If a person’s hands are hidden behind their body, are they holding something? The honest answer may be that it cannot be determined ; but if the project only allows “yes” or “no,” the annotator is forced to guess.

A better structure gives annotators more room to be accurate:

• Present ; clearly visible and confirmed
• Absent ; clearly not there
• Occluded ; hidden by something in the scene
• Unclear ; visible but too ambiguous to judge confidently
• Not visible ; outside the frame or view entirely

This matters enormously for things like object possession, safety-gear checks, facial attributes, and partially visible hand movements. Forcing people to guess does not make a dataset more complete  it simply turns honest uncertainty into confident-looking noise. That kind of noise is especially difficult to catch later because, on the surface, the annotation looks perfectly finished.

9. Image vs. Video: Two Different Questions, Not Two Different File Types

The difference between image annotation for security and surveillance and video annotation for security and surveillance is not really about whether the file is a picture or a clip. It is about the kind of question the model needs to answer.

Image annotation works well when the system only needs to know things like:

• Is a person or vehicle present?
• Is a bag visible?
• Is safety gear being worn?
• Where does the restricted zone sit?
• Is a license plate visible, and where?

Video annotation becomes necessary when the system needs to know things like:

• Did the person cross a boundary?
• How long did they stay there?
• Is this the same person as a moment ago?
• Did they leave an object behind, and come back for it?
• Did two people enter together?
• Exactly when did the event begin and end?

The simplest way to think about it is this: if the model needs information from earlier frames to make the right decision in the current frame, image annotation alone will not be enough. Most real surveillance systems end up combining both ; images teach the system what exists, and video teaches it how those things behave over time.

10. Why a One-Line Definition Never Survives Production

A guideline that simply says “intrusion equals a person entering a restricted area” is not really a complete guideline; it is just a definition. Making it usable in production means adding enough detail to remove room for personal interpretation, and a mature guideline usually covers five distinct layers:

• Object rules – minimum visibility, partial or distant subjects, occlusion handling
• Zone rules – exact geometry, perspective handling, what counts as entry or exit
• Tracking rules – how long an ID persists, how re-entry is handled, when a track ends or restarts
• Event rules – preconditions, start and end frames, duration, multi-person interactions
• Ambiguity rules – unknown states, what shouldn’t be labelled at all, when to escalate to a reviewer

These guidelines work best when they are built around difficult, real footage rather than clean, simple examples. That is why we always run a small pilot round before full production starts:

Pilot batch → disagreement review → guideline update → qualification round → production → ongoing calibration

The first version of a guideline is almost never the final one. New footage continues to reveal situations the original document did not cover, and the workflow needs to absorb those cases instead of leaving every annotator to solve them independently.

11. How We Check Quality at Pixel Annotation: Beyond the Bounding Box

Traditional annotation QA still matters here ; checking bounding-box accuracy, segmentation accuracy, classification accuracy, and attribute accuracy all remain part of the baseline. But at Pixel Annotation, we’ve learned that a surveillance model trained to recognise events, not just objects, needs another layer of quality review on top of that. For every security project, our QA also tracks:

• Track continuity across full sequences, not single frames
• How often IDs switch unexpectedly
• Agreement on zone crossings between annotators
• Agreement on dwell time
• Agreement on the event class itself
• Agreement on the exact start and end frames

Here’s the part that is easy to miss, and the part our review process is built around: two annotators can draw nearly identical boxes around the same person, agree closely on the shape of the restricted zone, and even maintain the same tracking ID throughout ; and still disagree on the exact frame where the intrusion actually starts. If the model is being trained on those temporal boundaries, that disagreement matters just as much as a missed box.

This is exactly why we don’t stop at “did both annotators draw the box correctly?” We also ask whether both annotators made the same security decision. For event-based datasets, that second question is usually the more meaningful measure of quality ; and it is the standard we apply in our own QA process.

12. Turning False Alarms Into Better Training Data

Annotation work does not need to stop after the first model is deployed. In many production workflows, the next and better version of the dataset comes directly from the model’s own mistakes. If a system keeps flagging any of the following, each of those clips is worth bringing back into the dataset:

• Workers walking near a boundary without crossing it
• Sunlight reflecting off wet pavement
• Delivery drivers waiting patiently at a door
• Bags set down for just a moment
• Coworkers entering together with proper authorisation
• People who briefly disappeared behind equipment rather than actually leaving

A practical improvement loop tends to look like this:

Model prediction → false-alert review → edge-case collection → guideline clarification → targeted annotation → retraining

This is usually more valuable than simply adding more random surveillance footage to the dataset, because the model is effectively showing the annotation team exactly which situations it still does not understand ; and those are the cases that should guide what gets labelled next.

The Difficult Label Is Not “Person.” It Is “Event.”

For anyone seeing surveillance annotation for the first time, it can look like a fairly straightforward computer vision problem: find the person, find the vehicle, find the bag, mark the restricted area. Those labels absolutely matter, but they are only the foundation. The real complexity begins when the model needs to understand how all of those things relate to each other:

• Walking near a boundary, versus actually crossing it
• Waiting, versus loitering
• A temporary placement, versus a genuine abandonment
• Occlusion, versus disappearance
• Permitted group entry, versus tailgating
• Ordinary, everyday movement, versus something genuinely worth an alert

That is ultimately what data annotation for security and surveillance provides ; structured ground truth not just for what a camera sees, but for what that activity actually means as it develops over time. For a non-technical reader, that is the main idea worth taking away. For a technical team, the harder work sits just underneath it: stable tracks, spatial rules, temporal boundaries, well-chosen hard negatives, honest uncertainty states, meaningful agreement metrics, and a feedback loop built around real model failures.

The camera captures the scene. The annotation is what teaches the model how to understand it.

At Pixel Annotation, this is the kind of ground-truth work our in-house teams handle every day ; building the guidelines, running the quality checks, and closing the loop between models in the real world and the situations they still get wrong.