----------------------- REVIEW 1 ---------------------
PAPER: 1
TITLE: Using machine learning to predict temporal orientation of search engines' queries in the Temporalia challange
AUTHORS: Michele Filannino and Goran Nenadic


----------- REVIEW -----------
Dear NTCIR-11 Temporalia Participants,

Thank you again for your participation to the NTCIR-11 Temporalia task and submission of participant paper. The following contains some general comments to all participants, followed by specific comments to your paper. Please read them both carefully, and make necessarily changes to your initial submission. The deadline for the camera-ready copy is November 1st. The submission URL is the same as before. Feel free to contact us If you have any questions regarding the correction.

See https://sites.google.com/site/ntcirtemporalia/submission or http://ntcir.nii.ac.jp/Temporalia/NTCIR-11-Temporalia/Submission/ for submission details.

Thanks,

Hideo, Adam, and Roi


---
General comments
---
* Please run spell checkers again to make sure there is not typos and grammatical errors.
* Please refer to our overview paper for the detail description of Temporalia task.

Joho, H., Jatowt, A., Blanco. R., Naka, H., and Yamamoto, S.
Overview of NTCIR-11 Temporal Information Access (Temporalia) Task
In: Proceedings of the NTCIR-11 Conference, Tokyo, Japan.
2014.

* If you used our document collection in TIR subtask, please refer to the following paper about the construction of the corpus.

Matthews, M., Tolchinsky, P., Blanco, R., Atserias, J., Mika, P., and Zaragoza, H.
Searching Through Time in the New York Times.
In: Proceedings of the 4th Workshop on Human-Computer Interaction and Information Retrieval, NJ, USA, pp.41-44.
2010.


---
Specific comments
---
The authors propose classification approach for TQIC subtask. Keeping in mind the small dataset size (dry run dataset) they engineer features so as to minimize feature sparsity.
19 features are used where some are computed from query, some derived from submission date and some from both. For feature computation the authors use POS tagger, TempoWordnet and comparison with Wikipedia titles.

Experiments are conducted with SVM equiped with linear, polynomial and RBF kernels, NB, C4.5 and Random Forest. The authors construct 3 runs with different feature sets, and the minimal feature set works best. Although they achieved 66% accuracy in formal runs they lifted it to up to 76% with a-posteriori improvements.

The authors could provide better motivation for using feature 1. They should also better describe Tempowordnet.

Formatting is OK, but few typos need to be corrected as follows.
- change to "queries" in caption of Table 1
-Yahoo! Research labs is not official organizer of Tenporalia (one organizer is affiliated with that institution)
-change to "cardinality of the value" in caption of Table 2
-"The task can natrally be seen as a" in Sec. 3
-"We mostly focussed our work on designing and testing" in Sec. 3
-"Attribute 10 and 15 are the most frequent " in Sec. 3.2
-Please finish the sentence "The rationALITY behind the use of POS is that,..." at the end of Sec. 3.2
-"has been illustrated in Sec. 3.2" in Conclusions

The paper also lacks reference to the task overview paper. Please add it.
---