CI7521 Assignment 1: Classical Machine Learning 2024-2025 |

Overview

This assignment counts for 50% of the overall mark for this module. Its subject is implementing Classical Machine Learning solutions in Python using the Scikit-Learn library and other libraries introduced in the class. Specifically, classification methods should be applied to the Digits Dataset:

https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html#sklearn.datasets.load_digits

This is a group assignment that includes both group and individual submissions as well as self-assessment. Marking will also consider how the group members rate the contributions of each other. You can form groups of up to four students.

Deliverables and Submission

The coursework must be submitted by the deadline given on Canvas. Follow the submission guidelines in Canvas. For each submission ensure that you include

• (Group Submission) A single IPython notebook (.ipynb) containing all code, text (including references) and results (see Part I below). Any results presented should be directly reproducible from the code without any modification. 
• (Group Submission) The IPython notebook must be converted to a word file which should also be submitted. Ensure that all code, text and results are included in the word file. If generative AI tools were used for producing the code, an appropriate Appendix must be added at the end of the document. Guidance on how to convert ipynb to docx is given on Canvas.
• (Individual Submission) Report with a specific structure (see below in Part II) in word or pdf format with the student’s name, k-number and group at the beginning of the document.

Rules

• If you have re-used large chunks of code from third-party sources, clearly specify the chunk (e.g. comment in the relevant cell), cite the source(s) and add references at the end of the notebook.
• Use of generative AI tools is permissible, but you should fully acknowledge it, as described in the University guidance at AI and
• Assessment: Digital Learning & Tools. In addition, you are expected to provide appendices with a full record of prompts and responses you have received from generative AI tools for each submission (group/individual).
• Usage of any third-party libraries that have not been used in the class must be explicitly approved by the Lecturer by email.
• In case the above rules are not obeyed, marks may be deducted and/or the submission may be considered for plagiarism and penalised according to the University regulations.

Project Parts

PART I – Application: Classification: Training and Testing (Group Submission)

a)    The notebook must be strictly based on the template provided on Canvas.

b)    You should not load data from csv files or external websites. The specific dataset is available directly from the scikit-learn interface, see details at: https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html#sklearn.datasets.load_digits

c)    You should use at least 3 (but ideally up to 8) classification methods to distinguish between the classes, (see marking scheme). You are encouraged to look in the literature and identify methods beyond the ones mentioned in the module. In this case, you should reference the source.  Using a method with and without dimensionality reduction counts as multiple methods; ensure that any variations are listed separately.  
d) The following training/testing protocol should be used for all methods:

• Split the data into training (80%) and testing (20%).

e)  Evaluate all classification approaches using appropriate metrics such as

• Balanced Accuracy,
• ROC AUC (using the macro-averaged approach).

In addition,

•  Draw the macro-averaged ROCs for all classification methods into a single graph to allow for easy comparison between methods. You are not expected to produce ROC curves for individual classes (digits).
• Draw all relevant confusion matrices.

f) Consider and implement hyper-tuning of the parameters of your classification methods that could further improve the results. In this case, results for both the default and the optimal set of parameters should be provided.

g) The code for training the methods and the code for evaluating the methods should be in separate sections (see notebook template).

h)  When producing results, use appropriately the print() method or labels for tables and figures, so the meaning of results is clear without looking at the code.

i)  References (in text) are expected at the end of the notebook.

j) The word document should also include an Appendix on the use of generative AI tools. Provide a full record of prompts and responses used for producing the code, if applicable

PART II – Report (Individual Submission):

This is an individual submission (word or pdf), so you should not collaborate or exchange your views with the other members of your group. The report must strictly follow the structure below, and must not be more than three pages, including text and figures, but excluding references and appendix:

a.  Methods: Briefly list the methods you used and cite the original academic paper, where possible, otherwise cite an appropriate academic source (e.g. textbook). DO NOT explain the methods and DO NOT cite code.
b. Results: Explain the parameters used in your experiments. You can present results differently here to support your critical arguments and conclusions. 
c. Discussion/Conclusion: Provide up to three conclusions as a list of bullet points that are directly and logically derived from the discussion above. One of the conclusions must be your recommendation of a single method and parameter configuration. Conclusions such as “the second recommended method is…”  or “the worst method is…” ) are not acceptable though. Each conclusion should consist of 

i.    a conclusion outline, i.e. a single sentence that uses the following schema:{conclusion statement}, “, as seen in   ” {reference to evidence such as table/figure},  

ii.    a short critical discussion (few sentences) to clarify the conclusion and potentially link it to theory.

b)  References:  List any references you may have used in your document and any possible use of generative AI, using one of the established referencing systems (e.g. IEEE, Harvard, etc).

c)  Appendix I: Use the template in Appendix I. If you used generative AI tools, provide a full record of prompts and responses used for producing this report, if applicable.

d)  Self-evaluation: Use the template in APPENDIX II to assess both the group submission and your own individual submission, as well as to rate the contributions of the group members towards the group submission.

Marks will be deducted if the report does not follow the proposed structure. Also, any material beyond the page limits will NOT be considered.
 

Learning outcomes being assessed

• Select and specify suitable methods and algorithms relevant for a particular data analysis process;
• Develop machine learning and artificial intelligence systems using software packages and/or specialised libraries;
• Articulate and demonstrate the specific problems associated with different phases or tasks of a machine learning or artificial intelligence pipeline;
• Assess and evaluate machine learning methods using datasets and appropriate criteria.