Social competence survey for bonus points #839

juandpenan · 2023-12-18T09:30:38Z

** Note: Your contribution is expected to meet the conventions and policies described in the contribution guidelines **

Description

Closes issue #827

Changes proposed in this pull request:

Introduced bonuses for receptionist and restaurant tasks.
Assigned qualitative assessment scores ranging from 0 to 50 based on a survey for teams.
Mandated each team to bring a minimum of two public evaluators and inform the referee beforehand.
Provided a Microsoft Form for surveys and an online spreadsheet for result visibility.

johaq

Like the idea. Couple of notes:

I'm not sure we want the referee to evaluate since they should be focusing on running the test
Probably need to keep the group of volunteers consistent for one test block for fairness in evaluation

Aside from the notes please change task to test which is the usual terminology in the rulebook (I know not 100% consistent everywhere but it should be)

general_rules/PenaltiesBonuses.tex

juandpenan · 2024-01-12T19:02:20Z

Like the idea. Couple of notes:

I'm not sure we want the referee to evaluate since they should be focusing on running the test

Probably need to keep the group of volunteers consistent for one test block for fairness in evaluation

Aside from the notes please change task to test which is the usual terminology in the rulebook (I know not 100% consistent everywhere but it should be)

Thank you, @johaq, for the review! I've incorporated the suggested changes and added an additional item based on our discussion in the last meeting. " The referee has the authority to skip the social assessment test if they believe the robot's performance is not suitable for measurement."

johaq

Two small typos.
Otherwise looks good to me.

general_rules/PenaltiesBonuses.tex

juandpenan · 2024-01-12T20:13:14Z

Two small typos. Otherwise looks good to me.

Done! Thanks a lot

akinobu1998 · 2024-01-20T02:00:50Z

The form and spreadsheet look like owned by a personal account and the spreadsheet is private.
Is this URL and QR code the finalized version? If not, we can distribute the QR code via Telegram or printed QR code.
Sometimes online document disappears by accident, and I also worry about spam. Then we had to change the link.

I think we should also include the question list in the rulebook to track changes and review the questions more easily.

johaq · 2024-01-20T08:55:58Z

I'm not familiar with office forms and if you can host them on your own. Ideally the form would be on the athome website or hosted by a University I think.

LeroyR · 2024-02-28T06:50:49Z

general_rules/PenaltiesBonuses.tex

@@ -42,6 +42,26 @@ \subsection{Bonus for outstanding performance}\label{rule:outstanding_performanc
 	\item It is the decision of the \iaterm{Technical Committee}{TC} if (and to which degree) the bonus score is granted.
 \end{enumerate}

+\subsection{Bonus for perceived social intelligence}\label{rule:perceived_intelligence}
+\begin{enumerate}	
+    \item For the test \iterm{Receptionist} in \iterm{Stage~I} and, \iterm{Restaurant} in \iterm{Stage~II} tests. Teams are allowed to request an assessment of the robot's performance regarding its perceived social intelligence.


Imho should be mandatory and integrated in the scoring of the tasks - to make sure league/referee prepares volunteers.
To be useable data we have to define a minimum number of respondents,
for testing the procedure (or getting an overview of the league) in local leagues we can always opt to skip parts of a task e.g. this survey.

I don't think it has to be mandatory, but I agree, that it should be integrated in the scoring of the tasks. Since you can't lose anything anyway, any rational team should always opt for social scoring.

Yes, I think we can just make it mandatory

~~Teams are allowed~~ Teams are assessed

LeroyR · 2024-02-28T06:53:31Z

general_rules/PenaltiesBonuses.tex

+
+    \item Every team seeking a social assessment should inform the referee before the test. that they want to be evaluated using perceived social intelligence.    
+
+    \item After the test is completed, the evaluators will fill out the form accessible via the QR code in Figure \ref{fig:qr-survey}, or through this \href{https://forms.office.com/Pages/ResponsePage.aspx?id=6sSEXw03nkuDDHVvi_G1H7VNGCdGFtZJs0ryJVVWtCFUQVFSWDlYM0FHRVA2QllIT0tOQjI2QUcxQi4u}{link}.


Need to be hosted somewhere under control of the league. Fallback sheet in this github for printing.

We can figure out hosting later

LeroyR · 2024-02-28T06:53:43Z

general_rules/PenaltiesBonuses.tex

+
+    \item The referee has the authority to skip the social assessment test if they believe the robot's performance is not suitable for measurement.
+
+    \item The score will be automatically recorded in this \href{https://urjc-my.sharepoint.com/:x:/r/personal/juan_pena_urjc_es/Documents/ROBOCUP%20@HOME%20PSI%20SCALE%20PROPOSAL.xlsx?d=wfdc816bee34742e1a9e5bea95677985d&csf=1&web=1&e=zRwl4u}{spreadsheet}.


Need to be hosted somewhere under control of the league. Fallback script maybe? Link or document how to calculate score atleast.

I also think we need to mention how the score is calculated. Average value from all 15 ratings collected times 10, so that a maximum of 50 points can be achieved.

$$\text{Social Score} = \text{scaling} \times \frac{1}{n} \sum_{i=1}^{n} \text{Rating}_i$$

$$\text{scaling} = \frac{\text{max Social Score}}{\text{max Rating}} = 10$$

$$\text{n} = | \text{{Rating}} |$$

LeanderVonSeelstrang

Generally good, but it should be fully integrated into the task and the method of converting the assessment into a score should be explicitly stated. Also the survey should always be linked. Possibly a non-editable version.

general_rules/PenaltiesBonuses.tex

LeanderVonSeelstrang · 2024-02-28T22:11:20Z

general_rules/PenaltiesBonuses.tex

+
+    \item This bonus, ranging from 0 to 50, depends on the robot's social performance which will be assessed by Referees using a specially designed scale in a survey.
+
+    \item Every team seeking a social assessment should inform the referee before the test. that they want to be evaluated using perceived social intelligence.    


Small typo. A point in the middle after test. Also should -> must. Maybe also request how mich before. E.g. an hour.

Any team seeking a social assessment must inform the referee at least one hour before the test that they wish to be evaluated on perceived social intelligence.

Should actually be the day before the test in the team leader meeting

general_rules/PenaltiesBonuses.tex

LeanderVonSeelstrang · 2024-02-28T22:34:38Z

general_rules/PenaltiesBonuses.tex

+
+    \item The referee has the authority to skip the social assessment test if they believe the robot's performance is not suitable for measurement.
+
+    \item The score will be automatically recorded in this \href{https://urjc-my.sharepoint.com/:x:/r/personal/juan_pena_urjc_es/Documents/ROBOCUP%20@HOME%20PSI%20SCALE%20PROPOSAL.xlsx?d=wfdc816bee34742e1a9e5bea95677985d&csf=1&web=1&e=zRwl4u}{spreadsheet}.


I also think we need to mention how the score is calculated. Average value from all 15 ratings collected times 10, so that a maximum of 50 points can be achieved.

$$\text{Social Score} = \text{scaling} \times \frac{1}{n} \sum_{i=1}^{n} \text{Rating}_i$$

$$\text{scaling} = \frac{\text{max Social Score}}{\text{max Rating}} = 10$$

$$\text{n} = | \text{{Rating}} |$$

LeanderVonSeelstrang · 2024-02-28T22:40:50Z

general_rules/PenaltiesBonuses.tex

@@ -42,6 +42,26 @@ \subsection{Bonus for outstanding performance}\label{rule:outstanding_performanc
 	\item It is the decision of the \iaterm{Technical Committee}{TC} if (and to which degree) the bonus score is granted.
 \end{enumerate}

+\subsection{Bonus for perceived social intelligence}\label{rule:perceived_intelligence}
+\begin{enumerate}	
+    \item For the test \iterm{Receptionist} in \iterm{Stage~I} and, \iterm{Restaurant} in \iterm{Stage~II} tests. Teams are allowed to request an assessment of the robot's performance regarding its perceived social intelligence.


I don't think it has to be mandatory, but I agree, that it should be integrated in the scoring of the tasks. Since you can't lose anything anyway, any rational team should always opt for social scoring.

…oring-survey

…staurant test Signed-off-by: juan <[email protected]>

juandpenan · 2024-03-04T15:42:57Z

Thanks for the feedback. So I modify the following:

Added questions
Added the score calculation method
Made the score "mandatory" for receptionist and restaurant tests
Also in the upcoming days will look for different options to host the survey.

Thanks!

LeroyR

lgtm, slight wording change needed as assessment is mandatory

LeroyR · 2024-03-04T19:37:12Z

general_rules/PenaltiesBonuses.tex

@@ -42,6 +42,26 @@ \subsection{Bonus for outstanding performance}\label{rule:outstanding_performanc
 	\item It is the decision of the \iaterm{Technical Committee}{TC} if (and to which degree) the bonus score is granted.
 \end{enumerate}

+\subsection{Bonus for perceived social intelligence}\label{rule:perceived_intelligence}
+\begin{enumerate}	
+    \item For the test \iterm{Receptionist} in \iterm{Stage~I} and, \iterm{Restaurant} in \iterm{Stage~II} tests. Teams are allowed to request an assessment of the robot's performance regarding its perceived social intelligence.


~~Teams are allowed~~ Teams are assessed

LeanderVonSeelstrang

Thanks for the update.

juandpenan · 2024-03-05T08:17:24Z

lgtm, slight wording change needed as assessment is mandatory

Done! thankss

psi added as a bonus

2ab3653

johaq requested changes Jan 5, 2024

View reviewed changes

general_rules/PenaltiesBonuses.tex Outdated Show resolved Hide resolved

general_rules/PenaltiesBonuses.tex Outdated Show resolved Hide resolved

general_rules/PenaltiesBonuses.tex Outdated Show resolved Hide resolved

Update PenaltiesBonuses.tex, answering to the review comments

b3b3682

johaq requested changes Jan 12, 2024

View reviewed changes

general_rules/PenaltiesBonuses.tex Outdated Show resolved Hide resolved

general_rules/PenaltiesBonuses.tex Outdated Show resolved Hide resolved

Update PenaltiesBonuses.tex

161c9e9

johaq previously approved these changes Jan 13, 2024

View reviewed changes

Merge branch 'master' into fix/RoboCupAtHome#827-incorporing-survey

1e4bcda

sunava self-requested a review February 26, 2024 12:08

johaq requested a review from LeanderVonSeelstrang February 26, 2024 12:08

LeroyR requested changes Feb 28, 2024

View reviewed changes

LeanderVonSeelstrang requested changes Feb 28, 2024

View reviewed changes

juandpenan added 4 commits March 4, 2024 10:43

Merge branch 'RoboCupAtHome:master' into fix/RoboCupAtHome#827-incorp…

b597171

…oring-survey

Merge branch 'RoboCupAtHome:master' into fix/RoboCupAtHome#827-incorp…

7af286f

…oring-survey

form questions, score formula, and mandatory for receptionist and re…

7d7919d

…staurant test Signed-off-by: juan <[email protected]>

small fix

8deb3d0

juandpenan dismissed johaq’s stale review via 8deb3d0 March 4, 2024 15:36

LeroyR requested review from LeroyR, johaq and LeanderVonSeelstrang March 4, 2024 19:31

LeroyR previously approved these changes Mar 4, 2024

View reviewed changes

LeanderVonSeelstrang previously approved these changes Mar 4, 2024

View reviewed changes

Making it clear, evaluation is mandatory in the bonus description.

ed2d76b

juandpenan dismissed stale reviews from LeanderVonSeelstrang and LeroyR via ed2d76b March 5, 2024 08:16

johaq approved these changes Mar 5, 2024

View reviewed changes

LeroyR removed the request for review from sunava March 5, 2024 09:20

LeroyR merged commit 01eecfe into RoboCupAtHome:master Mar 5, 2024
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Social competence survey for bonus points #839

Social competence survey for bonus points #839

juandpenan commented Dec 18, 2023

johaq left a comment

juandpenan commented Jan 12, 2024 •

edited

Loading

johaq left a comment

juandpenan commented Jan 12, 2024

akinobu1998 commented Jan 20, 2024

johaq commented Jan 20, 2024

LeroyR Feb 28, 2024

LeanderVonSeelstrang Feb 28, 2024

johaq Feb 29, 2024

LeroyR Mar 4, 2024

LeroyR Feb 28, 2024

johaq Mar 3, 2024

LeroyR Feb 28, 2024

LeanderVonSeelstrang Feb 28, 2024

LeanderVonSeelstrang left a comment

LeanderVonSeelstrang Feb 28, 2024

johaq Feb 29, 2024

LeanderVonSeelstrang Feb 28, 2024

LeanderVonSeelstrang Feb 28, 2024

juandpenan commented Mar 4, 2024

LeroyR left a comment

LeroyR Mar 4, 2024

LeanderVonSeelstrang left a comment

juandpenan commented Mar 5, 2024


		\item Every team seeking a social assessment should inform the referee before the test. that they want to be evaluated using perceived social intelligence.

		\item After the test is completed, the evaluators will fill out the form accessible via the QR code in Figure \ref{fig:qr-survey}, or through this \href{https://forms.office.com/Pages/ResponsePage.aspx?id=6sSEXw03nkuDDHVvi_G1H7VNGCdGFtZJs0ryJVVWtCFUQVFSWDlYM0FHRVA2QllIT0tOQjI2QUcxQi4u}{link}.


		\item The referee has the authority to skip the social assessment test if they believe the robot's performance is not suitable for measurement.

		\item The score will be automatically recorded in this \href{https://urjc-my.sharepoint.com/:x:/r/personal/juan_pena_urjc_es/Documents/ROBOCUP%20@HOME%20PSI%20SCALE%20PROPOSAL.xlsx?d=wfdc816bee34742e1a9e5bea95677985d&csf=1&web=1&e=zRwl4u}{spreadsheet}.


		\item This bonus, ranging from 0 to 50, depends on the robot's social performance which will be assessed by Referees using a specially designed scale in a survey.

		\item Every team seeking a social assessment should inform the referee before the test. that they want to be evaluated using perceived social intelligence.

Social competence survey for bonus points #839

Social competence survey for bonus points #839

Conversation

juandpenan commented Dec 18, 2023

Description

johaq left a comment

Choose a reason for hiding this comment

juandpenan commented Jan 12, 2024 • edited Loading

johaq left a comment

Choose a reason for hiding this comment

juandpenan commented Jan 12, 2024

akinobu1998 commented Jan 20, 2024

johaq commented Jan 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LeanderVonSeelstrang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juandpenan commented Mar 4, 2024

LeroyR left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LeanderVonSeelstrang left a comment

Choose a reason for hiding this comment

juandpenan commented Mar 5, 2024

juandpenan commented Jan 12, 2024 •

edited

Loading