Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Social competence survey for bonus points #839

Merged
merged 9 commits into from
Mar 5, 2024
Merged

Social competence survey for bonus points #839

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
2ab3653
psi added as a bonus
juandpenan Dec 17, 2023
b3b3682
Update PenaltiesBonuses.tex, answering to the review comments
juandpenan Jan 12, 2024
161c9e9
Update PenaltiesBonuses.tex
juandpenan Jan 12, 2024
1e4bcda
Merge branch 'master' into fix/#827-incorporing-survey
johaq Jan 14, 2024
b597171
Merge branch 'RoboCupAtHome:master' into fix/#827-incorporing-survey
juandpenan Mar 4, 2024
7af286f
Merge branch 'RoboCupAtHome:master' into fix/#827-incorporing-survey
juandpenan Mar 4, 2024
7d7919d
form questions, score formula, and mandatory for receptionist and re…
juandpenan Mar 4, 2024
8deb3d0
small fix
juandpenan Mar 4, 2024
ed2d76b
Making it clear, evaluation is mandatory in the bonus description.
juandpenan Mar 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions general_rules/PenaltiesBonuses.tex
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,26 @@ \subsection{Bonus for outstanding performance}\label{rule:outstanding_performanc
\item It is the decision of the \iaterm{Technical Committee}{TC} if (and to which degree) the bonus score is granted.
\end{enumerate}

\subsection{Bonus for perceived social intelligence}\label{rule:perceived_intelligence}
\begin{enumerate}
\item For the test \iterm{Receptionist} in \iterm{Stage~I} and, \iterm{Restaurant} in \iterm{Stage~II} tests. Teams are allowed to request an assessment of the robot's performance regarding its perceived social intelligence.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Imho should be mandatory and integrated in the scoring of the tasks - to make sure league/referee prepares volunteers.
To be useable data we have to define a minimum number of respondents,
for testing the procedure (or getting an overview of the league) in local leagues we can always opt to skip parts of a task e.g. this survey.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it has to be mandatory, but I agree, that it should be integrated in the scoring of the tasks. Since you can't lose anything anyway, any rational team should always opt for social scoring.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think we can just make it mandatory

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Teams are allowed Teams are assessed


\item This bonus, ranging from 0 to 50, depends on the robot's social performance which will be assessed by Referees using a specially designed scale in a survey.
LeroyR marked this conversation as resolved.
Show resolved Hide resolved

\item Every team seeking a social assessment should inform the referee before the test. that they want to be evaluated using perceived social intelligence.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small typo. A point in the middle after test. Also should -> must. Maybe also request how mich before. E.g. an hour.

Any team seeking a social assessment must inform the referee at least one hour before the test that they wish to be evaluated on perceived social intelligence.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should actually be the day before the test in the team leader meeting


\item After the test is completed, the evaluators will fill out the form accessible via the QR code in Figure \ref{fig:qr-survey}, or through this \href{https://forms.office.com/Pages/ResponsePage.aspx?id=6sSEXw03nkuDDHVvi_G1H7VNGCdGFtZJs0ryJVVWtCFUQVFSWDlYM0FHRVA2QllIT0tOQjI2QUcxQi4u}{link}.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to be hosted somewhere under control of the league. Fallback sheet in this github for printing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can figure out hosting later


\item The referee has the authority to skip the social assessment test if they believe the robot's performance is not suitable for measurement.
LeroyR marked this conversation as resolved.
Show resolved Hide resolved

\item The score will be automatically recorded in this \href{https://urjc-my.sharepoint.com/:x:/r/personal/juan_pena_urjc_es/Documents/ROBOCUP%20@HOME%20PSI%20SCALE%20PROPOSAL.xlsx?d=wfdc816bee34742e1a9e5bea95677985d&csf=1&web=1&e=zRwl4u}{spreadsheet}.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to be hosted somewhere under control of the league. Fallback script maybe? Link or document how to calculate score atleast.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also think we need to mention how the score is calculated. Average value from all 15 ratings collected times 10, so that a maximum of 50 points can be achieved.

$$\text{Social Score} = \text{scaling} \times \frac{1}{n} \sum_{i=1}^{n} \text{Rating}_i$$

$$\text{scaling} = \frac{\text{max Social Score}}{\text{max Rating}} = 10$$

$$\text{n} = | \text{{Rating}} |$$

\end{enumerate}

\begin{figure}[H]
\centering \includegraphics[width=0.3\linewidth]{images/QRCode for ROBOCUP @HOME PSI SCALE PROPOSAL(1).png}
\caption{QR code for accessing the survey}
\label{fig:qr-survey}
\end{figure}

% Local Variables:
% TeX-master: "../Rulebook"
Expand Down
Binary file added images/QRCode for ROBOCUP @HOME PSI SCALE PROPOSAL(1).png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.