In case of Derivational nouns, what is the difference between affix polysemy and affix homonymy?

The difference between polysemy and homonymy is often one of degree or the direction you approach them. They are difficult semantic relationships to fix with certainty even when it comes to lexical items let alone constructions with more abstract meanings (like tense or affixes).

But the examples you give are examples of polysemy NOT homonymy. You have one suffix -an with 2 related senses of relationship to root.

1. -an in American, African, Australian is not an example of affix polysemy. It is exactly the same affix applied to different words. The meaning of the affix being roughly 'inhabitant of or identifying as X'. Thus (with a bit of you get Martian, Venutian or even Xan.

2. -an in 'librarian' and 'comedian' has the meaning of 'related to X through the performance of a job'. You could argue, that there is some polysemy here as well, since a librarian does a job 'in' a library while a comedian 'performs' comedy. But it's a bit of a stretch.

A more clear-cut example of homonymy in suffixes is -s for plural as in 'books' and for possession as in 'Jake's' (in spoken English only). But this is for inflectional suffixes. I can't think of a derivational suffix homonymy in English but it is quite common in prefixes. For instance, the derivational prefix in- has completely unrelated meaning of 1. direction inward as in 'inhale', 'invite' and 2. negation 'invisible', 'inviolate', 'incredible'. Both of these unrelated meanings have multiple senses resulting in polysemy.