Predicting future trajectories of agents in a dynamic environment is essential for natural and safe decision making of autonomous agents. The trajectory of an agent in such an environment not only depends on the past motion of that agent but also depends on its interaction with other agents present in that environment. To capture the effect of other agents on trajectory prediction of an agent, we propose a three stream topology-aware graph convolutional network (TAGCN) for interaction message passing between the agents.  In addition, temporal encoding of local- and global-level topological features are fused to better characterize dynamic interactions between participants over time. Results are competitive compared to previous best methods for trajectory prediction on ETH and UCY datasets and highlights the need for both local and global interaction structure.