Section 01
Problem Statement
A young bricklayer receives a job on a high-rise balcony with an unusual joint pattern; a Spanish-speaking rebar worker cannot read the English method statement; a retiring foreman will lose his tacit knowledge in six months. The phone alone can teach the worker — with camera + ARCore + TFLite pose/segmentation for a visual overlay, speaker + SpeechRecognizer for a voice assistant that answers from local documents, video to capture the foreman's tricks, and a multilingual overlay (on-device or cloud translation for low latency). No AR/VR headset. No laptop.
Section 02
Solution Requirements
Phone AR task overlay
Using ARCore Geospatial/AugmentedImage, project the correct rebar layout, tie sequence, or panel snap-points onto the user's live camera view of a marked wall/corner, with a "next step / wrong" indicator plus a voice prompt.
Voice field assistant (on-device RAG)
Pressing a large button on a real ladder/scaffold triggers SpeechRecognizer → a retriever over 20 bundled method statements/specs → an answer returned via in-app TTS; offline fallback shows the top 3 hits if a full model isn't available. Accuracy on 20 golden questions ≥ 85%.
Video capture for the retiring expert
Record a video of a worker narrating a "trick of the trade"; extract key frames and auto-generate a "Lesson Card" with a short title (on-device NLP or cloud model), stored in a searchable library on the phone.
Multilingual worker interface
At minimum English + 2 other languages (e.g. Hindi + Spanish) via on-device translation (Firebase ML / ML Kit Translate); voice answers are translated into the target language and demoed live on the phone screen.
Progress / assessment record
After using the AR task, the worker answers 5 quick quiz questions; the app computes a % score and saves it to Room DB.
Accessibility
A high-contrast mode + TTS for visually-impaired users, vibration cues for hard-of-hearing users, and TalkBack labels for major UI elements, proven with a screen recording of TalkBack reading labels.
Offline training
AR steps and the 20 bundled documents work fully in airplane mode, including the quiz.
Section 03
Expected Solution / Outcomes
- The phone held up shows an AR wall rebar overlay + "Correct next step" on a real wall.
- On a ladder / real scaffold / porch, the worker asks a question by voice and receives a TTS answer.
- A recorded 60s trick-video is searchable afterward.
- Toggling the UI to Hindi shows a translated answer live.
Section 04
Technology & Hardware Constraints
Section 05
Scoring
Requirements Completeness
50
Each requirement (R-id) is worth 3–12 marks. A hard-coded or mocked claim earns 0 — only working code counts.
Presentation & Demo
50
3-slide pitch + video ≤ 7 min + live demo. Judges must understand the problem in ≤ 90 seconds.
Real-world Implementation & Feasibility
50
Would a PM, engineer, or worker install this with only a phone on Day 1 of the next project? Must work offline and cheaply. Field-trial videos prove real-environment execution (20+ pts of this score is the field video).
Innovation & Disruption
50
A novel fusion of phone sensors (e.g. IMU + camera + barometer = scaffold-instability detector) is rewarded over simple API-calling. Re-skinning an existing SaaS scores low.
Section 06
Bonus (+10)
A live phone-camera hand-recognition AR task: MediaPipe Hands recognises a rebar tie and counts ties vs. target, or a vibration/earpiece cue for the hard-of-hearing during an AR step.
Section 07
Deliverables
| Item | Format |
|---|---|
| Video Demo | ≤ 7 min, YouTube/Vimeo unlisted link. Must open with the phone alone on a table (no external hardware). |
| Field Trial Videos | (a) Real on-site/ladder/scaffold/porch AR overlay + voice Q&A with TTS. (b) A real recorded video of a family member/site friend (consent) telling a trick, then searched and played back. |
| Demo Link | Android APK download link (installed on the reviewer's own phone) or a mobile-optimised web app URL. |
| GitHub | arcore_module/, voice_assistant/, doc_corpus/, video_tricks/, translations/, models/ (TFLite). |
| Pitch Deck | 3-slide PDF (Problem → Solution → Field-Proof & Impact). |
| Technical Brief | 2-page PDF: architecture diagram, phone-sensor map, tech stack, limitations, PRIVACY.md. |
Section 08
Field-Trial Requirements
- Real AR overlay + voice Q&A demoed on-site or on a real ladder/scaffold.
- Real consented video of a worker's trick, recorded and searched back.
Section 09
Privacy Requirements
Any human face captured in a field video must be consented or blurred. A PRIVACY.md file is required in every repository. Consent is explicitly required for any recorded worker/colleague video.